2024-09-17T23:21:07.812 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T23:21:07.832 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T23:21:07.909 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909591 branch: squid description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_latest} workloads/compare-mirror-images-krbd} email: ceph-qa@ceph.com first_in_suite: false job_id: '7909591' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each name: teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.4: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.5: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.6: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f extra_system_packages: - pv selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: b0d356b98e08f6bc59ef15652f0d0e4298fc83c7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 1983 sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f sleep_before_teardown: 0 subset: 79/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_b0d356b98e08f6bc59ef15652f0d0e4298fc83c7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: b0d356b98e08f6bc59ef15652f0d0e4298fc83c7 targets: smithi096.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIvRkATk5dJFdg8bZ75kCPOVVti6XPuOAeNrFoSrWb1IZV+uOrgL1tk+67NoeOcH0c7OiLJ0627Yp39IrVU+X+A= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB07Wg4TIz69AdzDYYYnJEbgHMOwtweLxwBPUjQITs/NiMWDkBh0Ms3D/BGVdlo9PZk+OtslZcKDf/1fgkLbihA= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - workunit: clients: cluster1.client.mirror: - rbd/compare_mirror_images.sh env: RBD_DEVICE_TYPE: krbd timeout: 3h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_21:16:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T23:21:07.909 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_b0d356b98e08f6bc59ef15652f0d0e4298fc83c7/qa; will attempt to use it 2024-09-17T23:21:07.910 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_b0d356b98e08f6bc59ef15652f0d0e4298fc83c7/qa/tasks 2024-09-17T23:21:07.910 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T23:21:07.912 INFO:teuthology.task.internal:Checking packages... 2024-09-17T23:21:07.931 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c2ddf9721ef254645201cdbabd9c8f49c30a586f' 2024-09-17T23:21:07.931 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T23:21:07.931 INFO:teuthology.packaging:ref: None 2024-09-17T23:21:07.931 INFO:teuthology.packaging:tag: None 2024-09-17T23:21:07.932 INFO:teuthology.packaging:branch: squid 2024-09-17T23:21:07.932 INFO:teuthology.packaging:sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T23:21:07.932 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-17T23:21:08.158 INFO:teuthology.task.internal:Found packages for ceph version 19.2.0-0 2024-09-17T23:21:08.159 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T23:21:08.166 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T23:21:08.166 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T23:21:08.210 INFO:teuthology.task.internal:Saving configuration 2024-09-17T23:21:08.225 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T23:21:08.235 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T23:21:08.262 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909591', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 23:16:17.193246', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIvRkATk5dJFdg8bZ75kCPOVVti6XPuOAeNrFoSrWb1IZV+uOrgL1tk+67NoeOcH0c7OiLJ0627Yp39IrVU+X+A='} 2024-09-17T23:21:08.284 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909591', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-17 23:16:17.191430', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB07Wg4TIz69AdzDYYYnJEbgHMOwtweLxwBPUjQITs/NiMWDkBh0Ms3D/BGVdlo9PZk+OtslZcKDf/1fgkLbihA='} 2024-09-17T23:21:08.284 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T23:21:08.290 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-17T23:21:08.290 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-17T23:21:08.290 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T23:21:08.361 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7890bb7ac0>, signals=[15]) 2024-09-17T23:21:08.361 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T23:21:08.367 INFO:teuthology.task.internal:Opening connections... 2024-09-17T23:21:08.367 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2024-09-17T23:21:08.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T23:21:08.453 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-09-17T23:21:08.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T23:21:08.531 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T23:21:08.537 DEBUG:teuthology.orchestra.run.smithi096:> uname -m 2024-09-17T23:21:08.554 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2024-09-17T23:21:08.554 DEBUG:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2024-09-17T23:21:08.611 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Stream" 2024-09-17T23:21:08.611 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="9" 2024-09-17T23:21:08.611 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2024-09-17T23:21:08.611 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2024-09-17T23:21:08.611 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="9" 2024-09-17T23:21:08.611 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el9" 2024-09-17T23:21:08.611 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T23:21:08.611 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2024-09-17T23:21:08.611 INFO:teuthology.orchestra.run.smithi096.stdout:LOGO="fedora-logo-icon" 2024-09-17T23:21:08.611 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T23:21:08.611 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://centos.org/" 2024-09-17T23:21:08.612 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T23:21:08.612 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T23:21:08.612 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T23:21:08.612 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2024-09-17T23:21:08.634 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-09-17T23:21:08.653 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-09-17T23:21:08.653 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-09-17T23:21:08.710 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2024-09-17T23:21:08.710 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="9" 2024-09-17T23:21:08.710 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2024-09-17T23:21:08.711 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2024-09-17T23:21:08.711 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="9" 2024-09-17T23:21:08.711 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el9" 2024-09-17T23:21:08.711 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T23:21:08.711 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2024-09-17T23:21:08.711 INFO:teuthology.orchestra.run.smithi192.stdout:LOGO="fedora-logo-icon" 2024-09-17T23:21:08.711 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T23:21:08.711 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2024-09-17T23:21:08.711 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T23:21:08.711 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T23:21:08.711 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T23:21:08.711 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-09-17T23:21:08.734 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T23:21:08.743 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T23:21:08.802 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T23:21:08.802 DEBUG:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2024-09-17T23:21:08.805 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-09-17T23:21:08.821 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T23:21:08.826 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T23:21:08.826 DEBUG:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2024-09-17T23:21:08.862 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-09-17T23:21:08.916 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T23:21:08.964 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T23:21:08.976 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T23:21:08.977 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T23:21:08.977 DEBUG:teuthology.orchestra.run.smithi096:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T23:21:08.977 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T23:21:08.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T23:21:08.994 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-09-17T23:21:08.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T23:21:08.995 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-09-17T23:21:09.053 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-503.el9.x86_64 2024-09-17T23:21:09.054 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-503.el9.x86_64 2024-09-17T23:21:09.054 INFO:teuthology.task.kernel:Running kernel on smithi192: 5.14.0-503.el9.x86_64 2024-09-17T23:21:09.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum install -y kernel 2024-09-17T23:21:09.054 INFO:teuthology.task.kernel:Running kernel on smithi096: 5.14.0-503.el9.x86_64 2024-09-17T23:21:09.054 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum install -y kernel 2024-09-17T23:21:11.524 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.3 MB 00:00 2024-09-17T23:21:12.662 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.3 MB 00:01 2024-09-17T23:21:14.241 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 13 MB/s | 20 MB 00:01 2024-09-17T23:21:14.571 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-17T23:21:18.717 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.5 MB 00:00 2024-09-17T23:21:20.209 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 19 kB 00:00 2024-09-17T23:21:20.830 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.5 MB 00:02 2024-09-17T23:21:21.763 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-17T23:21:22.456 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 19 kB 00:00 2024-09-17T23:21:24.293 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 13 MB/s | 23 MB 00:01 2024-09-17T23:21:27.177 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-17T23:21:29.133 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T23:21:29.133 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T23:21:29.214 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-17T23:21:29.225 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-17T23:21:29.226 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-09-17T23:21:29.226 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-17T23:21:29.226 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-17T23:21:29.226 INFO:teuthology.orchestra.run.smithi096.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T23:21:29.226 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-17T23:21:29.226 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T23:21:29.226 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T23:21:29.226 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T23:21:29.226 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:21:29.226 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-17T23:21:29.226 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-17T23:21:29.227 INFO:teuthology.orchestra.run.smithi096.stdout:Install 4 Packages 2024-09-17T23:21:29.227 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:21:29.227 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 79 M 2024-09-17T23:21:29.227 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 126 M 2024-09-17T23:21:29.227 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-17T23:21:29.444 INFO:teuthology.orchestra.run.smithi096.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 300 kB/s | 36 kB 00:00 2024-09-17T23:21:29.708 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-09-17T23:21:30.028 INFO:teuthology.orchestra.run.smithi096.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-09-17T23:21:30.595 INFO:teuthology.orchestra.run.smithi096.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 27 MB/s | 35 MB 00:01 2024-09-17T23:21:30.970 INFO:teuthology.orchestra.run.smithi096.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 19 MB/s | 29 MB 00:01 2024-09-17T23:21:30.970 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:21:30.970 INFO:teuthology.orchestra.run.smithi096.stdout:Total 45 MB/s | 79 MB 00:01 2024-09-17T23:21:31.581 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T23:21:31.581 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T23:21:31.651 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-17T23:21:31.671 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-09-17T23:21:31.681 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-09-17T23:21:31.681 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-09-17T23:21:31.681 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-09-17T23:21:31.682 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-09-17T23:21:31.682 INFO:teuthology.orchestra.run.smithi192.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T23:21:31.682 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-09-17T23:21:31.682 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T23:21:31.682 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T23:21:31.682 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T23:21:31.682 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:21:31.682 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-09-17T23:21:31.682 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-09-17T23:21:31.682 INFO:teuthology.orchestra.run.smithi192.stdout:Install 4 Packages 2024-09-17T23:21:31.682 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:21:31.683 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 79 M 2024-09-17T23:21:31.683 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 126 M 2024-09-17T23:21:31.683 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-09-17T23:21:31.776 INFO:teuthology.orchestra.run.smithi192.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 533 kB/s | 36 kB 00:00 2024-09-17T23:21:32.038 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-17T23:21:32.038 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-17T23:21:32.560 INFO:teuthology.orchestra.run.smithi192.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-09-17T23:21:33.008 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-17T23:21:33.008 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-17T23:21:33.185 INFO:teuthology.orchestra.run.smithi192.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 20 MB/s | 29 MB 00:01 2024-09-17T23:21:33.552 INFO:teuthology.orchestra.run.smithi192.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 19 MB/s | 35 MB 00:01 2024-09-17T23:21:33.552 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:21:33.552 INFO:teuthology.orchestra.run.smithi192.stdout:Total 42 MB/s | 79 MB 00:01 2024-09-17T23:21:34.242 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-09-17T23:21:34.580 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-09-17T23:21:34.581 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-09-17T23:21:35.444 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-17T23:21:35.523 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-09-17T23:21:35.523 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-09-17T23:21:36.182 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T23:21:36.263 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T23:21:37.953 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-09-17T23:21:38.609 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T23:21:38.687 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T23:21:39.005 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T23:21:39.131 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T23:21:41.426 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T23:21:41.558 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T23:21:43.962 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T23:21:44.298 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T23:21:46.413 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T23:21:46.533 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T23:21:49.138 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T23:21:51.397 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T23:22:27.011 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T23:22:27.051 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T23:22:27.386 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T23:22:27.386 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T23:22:27.386 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T23:22:27.387 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T23:22:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T23:22:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:22:28.251 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-17T23:22:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T23:22:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T23:22:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T23:22:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T23:22:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:22:28.252 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-17T23:22:28.625 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T23:22:28.706 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T23:22:28.706 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T23:22:28.707 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T23:22:28.707 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T23:22:28.707 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T23:22:28.707 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-17T23:22:28.707 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi096.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T23:22:28.707 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T23:22:28.766 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T23:22:28.766 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T23:22:28.766 DEBUG:teuthology.orchestra.run.smithi096:> sudo rpm -qi grub2-tools 2024-09-17T23:22:28.798 INFO:teuthology.orchestra.run.smithi096.stdout:Name : grub2-tools 2024-09-17T23:22:28.798 INFO:teuthology.orchestra.run.smithi096.stdout:Epoch : 1 2024-09-17T23:22:28.798 INFO:teuthology.orchestra.run.smithi096.stdout:Version : 2.06 2024-09-17T23:22:28.798 INFO:teuthology.orchestra.run.smithi096.stdout:Release : 82.el9 2024-09-17T23:22:28.798 INFO:teuthology.orchestra.run.smithi096.stdout:Architecture: x86_64 2024-09-17T23:22:28.798 INFO:teuthology.orchestra.run.smithi096.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T23:22:28.798 INFO:teuthology.orchestra.run.smithi096.stdout:Group : Unspecified 2024-09-17T23:22:28.798 INFO:teuthology.orchestra.run.smithi096.stdout:Size : 8274670 2024-09-17T23:22:28.798 INFO:teuthology.orchestra.run.smithi096.stdout:License : GPLv3+ 2024-09-17T23:22:28.799 INFO:teuthology.orchestra.run.smithi096.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T23:22:28.799 INFO:teuthology.orchestra.run.smithi096.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T23:22:28.799 INFO:teuthology.orchestra.run.smithi096.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T23:22:28.799 INFO:teuthology.orchestra.run.smithi096.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T23:22:28.799 INFO:teuthology.orchestra.run.smithi096.stdout:Packager : builder@centos.org 2024-09-17T23:22:28.799 INFO:teuthology.orchestra.run.smithi096.stdout:Vendor : CentOS 2024-09-17T23:22:28.799 INFO:teuthology.orchestra.run.smithi096.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T23:22:28.799 INFO:teuthology.orchestra.run.smithi096.stdout:Summary : Support tools for GRUB. 2024-09-17T23:22:28.799 INFO:teuthology.orchestra.run.smithi096.stdout:Description : 2024-09-17T23:22:28.799 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:22:28.799 INFO:teuthology.orchestra.run.smithi096.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T23:22:28.799 INFO:teuthology.orchestra.run.smithi096.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T23:22:28.799 INFO:teuthology.orchestra.run.smithi096.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T23:22:28.800 INFO:teuthology.orchestra.run.smithi096.stdout:hardware devices. 2024-09-17T23:22:28.800 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:22:28.800 INFO:teuthology.orchestra.run.smithi096.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T23:22:28.800 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T23:22:28.801 INFO:teuthology.task.kernel:Updating grub on smithi096 to boot 5.14.0-508.el9.x86_64 2024-09-17T23:22:28.801 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T23:22:29.489 INFO:teuthology.orchestra.run.smithi096.stderr:Generating grub configuration file ... 2024-09-17T23:22:29.572 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T23:22:29.612 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T23:22:29.991 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T23:22:29.991 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T23:22:29.991 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T23:22:29.991 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T23:22:30.829 INFO:teuthology.orchestra.run.smithi096.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T23:22:30.871 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T23:22:30.871 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:22:30.871 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-09-17T23:22:30.871 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T23:22:30.871 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T23:22:30.871 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T23:22:30.871 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T23:22:30.872 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:22:30.872 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-09-17T23:22:30.872 INFO:teuthology.orchestra.run.smithi096.stderr:done 2024-09-17T23:22:30.875 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-09-17T23:22:30.891 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.k7gTsGndlp 2024-09-17T23:22:30.891 DEBUG:teuthology.orchestra.run.smithi096:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.k7gTsGndlp 2024-09-17T23:22:30.958 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.k7gTsGndlp 2024-09-17T23:22:31.108 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.k7gTsGndlp is 6KB 2024-09-17T23:22:31.160 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.k7gTsGndlp 2024-09-17T23:22:31.175 DEBUG:teuthology.orchestra.run.smithi096:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T23:22:31.206 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T23:22:31.245 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T23:22:31.245 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T23:22:31.245 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T23:22:31.245 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T23:22:31.246 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T23:22:31.286 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T23:22:31.286 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T23:22:31.287 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T23:22:31.287 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T23:22:31.287 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T23:22:31.287 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-17T23:22:31.287 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi192.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T23:22:31.287 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T23:22:31.345 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T23:22:31.346 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T23:22:31.346 DEBUG:teuthology.orchestra.run.smithi192:> sudo rpm -qi grub2-tools 2024-09-17T23:22:31.384 INFO:teuthology.orchestra.run.smithi192.stdout:Name : grub2-tools 2024-09-17T23:22:31.384 INFO:teuthology.orchestra.run.smithi192.stdout:Epoch : 1 2024-09-17T23:22:31.384 INFO:teuthology.orchestra.run.smithi192.stdout:Version : 2.06 2024-09-17T23:22:31.385 INFO:teuthology.orchestra.run.smithi192.stdout:Release : 82.el9 2024-09-17T23:22:31.385 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: x86_64 2024-09-17T23:22:31.385 INFO:teuthology.orchestra.run.smithi192.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T23:22:31.385 INFO:teuthology.orchestra.run.smithi192.stdout:Group : Unspecified 2024-09-17T23:22:31.385 INFO:teuthology.orchestra.run.smithi192.stdout:Size : 8274670 2024-09-17T23:22:31.385 INFO:teuthology.orchestra.run.smithi192.stdout:License : GPLv3+ 2024-09-17T23:22:31.385 INFO:teuthology.orchestra.run.smithi192.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T23:22:31.385 INFO:teuthology.orchestra.run.smithi192.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T23:22:31.385 INFO:teuthology.orchestra.run.smithi192.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T23:22:31.385 INFO:teuthology.orchestra.run.smithi192.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T23:22:31.385 INFO:teuthology.orchestra.run.smithi192.stdout:Packager : builder@centos.org 2024-09-17T23:22:31.386 INFO:teuthology.orchestra.run.smithi192.stdout:Vendor : CentOS 2024-09-17T23:22:31.386 INFO:teuthology.orchestra.run.smithi192.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T23:22:31.386 INFO:teuthology.orchestra.run.smithi192.stdout:Summary : Support tools for GRUB. 2024-09-17T23:22:31.386 INFO:teuthology.orchestra.run.smithi192.stdout:Description : 2024-09-17T23:22:31.386 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:22:31.386 INFO:teuthology.orchestra.run.smithi192.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T23:22:31.386 INFO:teuthology.orchestra.run.smithi192.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T23:22:31.386 INFO:teuthology.orchestra.run.smithi192.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T23:22:31.386 INFO:teuthology.orchestra.run.smithi192.stdout:hardware devices. 2024-09-17T23:22:31.386 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:22:31.386 INFO:teuthology.orchestra.run.smithi192.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T23:22:31.387 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T23:22:31.387 INFO:teuthology.task.kernel:Updating grub on smithi192 to boot 5.14.0-508.el9.x86_64 2024-09-17T23:22:31.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T23:22:31.394 DEBUG:teuthology.orchestra.run.smithi096:> sudo shutdown -r now 2024-09-17T23:22:32.077 INFO:teuthology.orchestra.run.smithi192.stderr:Generating grub configuration file ... 2024-09-17T23:22:33.355 INFO:teuthology.orchestra.run.smithi192.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T23:22:33.379 INFO:teuthology.orchestra.run.smithi192.stderr:done 2024-09-17T23:22:33.381 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-09-17T23:22:33.396 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.Nsl7LbmERP 2024-09-17T23:22:33.397 DEBUG:teuthology.orchestra.run.smithi192:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Nsl7LbmERP 2024-09-17T23:22:33.463 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.Nsl7LbmERP 2024-09-17T23:22:33.594 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.Nsl7LbmERP is 6KB 2024-09-17T23:22:33.642 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.Nsl7LbmERP 2024-09-17T23:22:33.657 DEBUG:teuthology.orchestra.run.smithi192:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T23:22:33.726 INFO:teuthology.orchestra.run.smithi192.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T23:22:33.726 INFO:teuthology.orchestra.run.smithi192.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T23:22:33.726 INFO:teuthology.orchestra.run.smithi192.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T23:22:33.726 INFO:teuthology.orchestra.run.smithi192.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T23:22:33.728 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T23:22:33.887 DEBUG:teuthology.orchestra.run.smithi192:> sudo shutdown -r now 2024-09-17T23:23:01.423 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-17T23:23:01.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-17T23:23:01.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T23:23:03.893 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-17T23:23:03.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-09-17T23:23:03.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T23:23:22.301 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-09-17T23:23:31.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-09-17T23:23:31.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T23:24:01.457 DEBUG:teuthology.orchestra.remote:timed out 2024-09-17T23:24:10.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-17T23:24:10.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T23:24:10.952 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-09-17T23:24:11.326 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-17T23:24:11.326 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T23:24:11.327 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-09-17T23:24:11.342 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-508.el9.x86_64 2024-09-17T23:24:11.342 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T23:24:11.342 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T23:24:11.342 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T23:24:12.343 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-17T23:24:12.343 DEBUG:teuthology.orchestra.run.smithi096:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T23:24:12.450 INFO:teuthology.orchestra.run.smithi096.stdout:ttyS1 2024-09-17T23:24:12.472 DEBUG:teuthology.parallel:result is None 2024-09-17T23:24:31.313 DEBUG:teuthology.orchestra.remote:timed out 2024-09-17T23:24:43.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-09-17T23:24:43.315 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T23:24:43.594 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-09-17T23:24:43.960 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-09-17T23:24:43.960 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T23:24:43.960 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-09-17T23:24:43.976 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-508.el9.x86_64 2024-09-17T23:24:43.976 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T23:24:43.976 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T23:24:43.976 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T23:24:44.977 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-17T23:24:44.977 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T23:24:45.047 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2024-09-17T23:24:45.067 DEBUG:teuthology.parallel:result is None 2024-09-17T23:24:45.067 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T23:24:45.075 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T23:24:45.075 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T23:24:45.078 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T23:24:45.096 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T23:24:45.117 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T23:24:45.167 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T23:24:45.167 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T23:24:45.170 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T23:24:45.205 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T23:24:45.216 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T23:24:45.216 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T23:24:45.238 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T23:24:45.267 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T23:24:45.275 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T23:24:45.279 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T23:24:45.287 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T23:24:45.289 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T23:24:45.295 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T23:24:45.295 DEBUG:teuthology.orchestra.run.smithi096:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T23:24:45.324 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T23:24:45.357 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T23:24:45.365 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T23:24:45.366 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T23:24:45.393 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T23:24:45.414 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T23:24:45.488 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T23:24:45.594 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T23:24:45.638 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T23:24:45.679 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:24:45.680 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T23:24:45.745 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T23:24:45.791 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T23:24:45.841 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T23:24:45.925 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T23:24:45.967 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:24:45.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T23:24:46.034 DEBUG:teuthology.orchestra.run.smithi096:> sudo service rsyslog restart 2024-09-17T23:24:46.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-09-17T23:24:46.083 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T23:24:46.119 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T23:24:46.552 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T23:24:46.559 INFO:teuthology.task.internal:Starting timer... 2024-09-17T23:24:46.560 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T23:24:46.604 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T23:24:46.654 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-17T23:24:46.654 DEBUG:teuthology.orchestra.run.smithi096:> sudo service auditd rotate 2024-09-17T23:24:46.706 INFO:teuthology.orchestra.run.smithi096.stdout:Rotating logs: 2024-09-17T23:24:46.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2024-09-17T23:24:46.766 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2024-09-17T23:24:46.768 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T23:24:46.768 DEBUG:teuthology.orchestra.run.smithi096:> /usr/sbin/getenforce 2024-09-17T23:24:46.803 INFO:teuthology.orchestra.run.smithi096.stdout:Permissive 2024-09-17T23:24:46.804 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2024-09-17T23:24:46.826 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2024-09-17T23:24:46.827 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi096.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2024-09-17T23:24:46.827 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-17T23:24:46.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T23:24:46.873 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-17T23:24:46.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T23:24:46.901 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T23:24:46.901 DEBUG:teuthology.orchestra.run.smithi096:> sudo /usr/sbin/setenforce permissive 2024-09-17T23:24:46.942 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2024-09-17T23:24:46.974 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T23:24:46.982 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-17T23:24:46.986 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-17T23:24:47.120 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T23:24:47.130 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T23:24:47.131 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi096.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T23:28:28.746 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi096.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-09-17T23:28:28.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-17T23:28:28.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T23:28:28.824 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-09-17T23:28:28.907 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-17T23:28:28.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-09-17T23:28:28.908 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T23:28:28.993 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-09-17T23:28:29.075 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-09-17T23:28:29.075 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T23:28:29.085 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T23:28:29.086 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T23:28:29.086 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T23:28:29.088 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T23:28:29.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T23:28:29.115 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T23:28:29.129 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T23:28:29.145 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T23:28:29.154 INFO:teuthology.orchestra.run.smithi096.stderr:sudo: ntpd: command not found 2024-09-17T23:28:29.157 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T23:28:29.164 INFO:teuthology.orchestra.run.smithi096.stdout:506 Cannot talk to daemon 2024-09-17T23:28:29.179 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T23:28:29.181 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2024-09-17T23:28:29.192 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2024-09-17T23:28:29.192 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T23:28:29.206 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T23:28:29.220 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T23:28:29.256 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-09-17T23:28:29.259 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T23:28:29.259 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-09-17T23:28:29.260 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:28:29.260 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:28:29.260 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:28:29.260 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:28:29.275 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-09-17T23:28:29.278 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T23:28:29.278 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-09-17T23:28:29.278 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:28:29.278 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:28:29.279 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:28:29.279 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T23:28:29.279 INFO:teuthology.run_tasks:Running task install... 2024-09-17T23:28:29.288 DEBUG:teuthology.task.install:project ceph 2024-09-17T23:28:29.288 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}, 'extra_system_packages': ['pv']} 2024-09-17T23:28:29.288 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f', 'extra_system_packages': ['pv']} 2024-09-17T23:28:29.288 INFO:teuthology.task.install:Using flavor: default 2024-09-17T23:28:29.293 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-17T23:28:29.293 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-17T23:28:29.294 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': ['pv'], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f', 'tag': None, 'wait_for_package': False} 2024-09-17T23:28:29.294 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T23:28:29.296 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': ['pv'], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f', 'tag': None, 'wait_for_package': False} 2024-09-17T23:28:29.296 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T23:28:29.438 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/ 2024-09-17T23:28:29.439 INFO:teuthology.task.install.rpm:Package version is 19.1.1-330.gc2ddf972 2024-09-17T23:28:29.446 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/ 2024-09-17T23:28:29.446 INFO:teuthology.task.install.rpm:Package version is 19.1.1-330.gc2ddf972 2024-09-17T23:28:29.558 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T23:28:29.558 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:28:29.558 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T23:28:29.561 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/c2ddf9721ef254645201cdbabd9c8f49c30a586f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-17T23:28:29.561 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:28:29.561 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-17T23:28:29.588 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror, pv, pv, pv on remote rpm x86_64 2024-09-17T23:28:29.588 DEBUG:teuthology.orchestra.run.smithi096:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c2ddf9721ef254645201cdbabd9c8f49c30a586f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T23:28:29.589 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror, pv, pv, pv, pv on remote rpm x86_64 2024-09-17T23:28:29.590 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c2ddf9721ef254645201cdbabd9c8f49c30a586f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-17T23:28:29.653 DEBUG:teuthology.orchestra.run.smithi096:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T23:28:29.655 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-17T23:28:29.729 DEBUG:teuthology.orchestra.run.smithi096:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T23:28:29.730 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-17T23:28:29.793 INFO:teuthology.orchestra.run.smithi096.stdout:check_obsoletes = 1 2024-09-17T23:28:29.795 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum clean all 2024-09-17T23:28:29.795 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2024-09-17T23:28:29.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2024-09-17T23:28:30.021 INFO:teuthology.orchestra.run.smithi096.stdout:45 files removed 2024-09-17T23:28:30.028 INFO:teuthology.orchestra.run.smithi192.stdout:45 files removed 2024-09-17T23:28:30.048 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd rbd-mirror pv pv pv pv 2024-09-17T23:28:30.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd rbd-mirror pv pv pv pv 2024-09-17T23:28:30.605 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 314 kB/s | 84 kB 00:00 2024-09-17T23:28:30.634 INFO:teuthology.orchestra.run.smithi096.stdout:ceph packages for x86_64 277 kB/s | 84 kB 00:00 2024-09-17T23:28:30.889 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-17T23:28:30.918 INFO:teuthology.orchestra.run.smithi096.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-17T23:28:31.173 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 7.6 kB/s | 1.9 kB 00:00 2024-09-17T23:28:31.193 INFO:teuthology.orchestra.run.smithi096.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-17T23:28:31.636 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.3 MB 00:00 2024-09-17T23:28:33.790 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.3 MB 00:02 2024-09-17T23:28:33.812 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-09-17T23:28:35.641 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-17T23:28:38.246 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-17T23:28:40.538 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-17T23:28:41.833 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.5 MB 00:02 2024-09-17T23:28:42.059 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-17T23:28:43.376 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 19 kB 00:00 2024-09-17T23:28:45.280 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 13 MB/s | 23 MB 00:01 2024-09-17T23:28:47.515 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-17T23:28:49.710 INFO:teuthology.orchestra.run.smithi096.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-17T23:28:49.711 INFO:teuthology.orchestra.run.smithi096.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-17T23:28:49.772 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-17T23:28:49.789 INFO:teuthology.orchestra.run.smithi096.stdout:====================================================================================== 2024-09-17T23:28:49.789 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-17T23:28:49.789 INFO:teuthology.orchestra.run.smithi096.stdout:====================================================================================== 2024-09-17T23:28:49.789 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-17T23:28:49.789 INFO:teuthology.orchestra.run.smithi096.stdout: ceph x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 6.5 k 2024-09-17T23:28:49.789 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 5.4 M 2024-09-17T23:28:49.789 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 897 k 2024-09-17T23:28:49.789 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 145 k 2024-09-17T23:28:49.789 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 1.5 M 2024-09-17T23:28:49.790 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 146 k 2024-09-17T23:28:49.790 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 3.6 M 2024-09-17T23:28:49.790 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 7.4 M 2024-09-17T23:28:49.790 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 49 k 2024-09-17T23:28:49.790 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 11 M 2024-09-17T23:28:49.790 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 49 M 2024-09-17T23:28:49.790 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 285 k 2024-09-17T23:28:49.790 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 765 k 2024-09-17T23:28:49.790 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 32 k 2024-09-17T23:28:49.790 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 751 k 2024-09-17T23:28:49.790 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 127 k 2024-09-17T23:28:49.790 INFO:teuthology.orchestra.run.smithi096.stdout: pv x86_64 1.6.20-1.el9 epel 65 k 2024-09-17T23:28:49.791 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 162 k 2024-09-17T23:28:49.791 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 320 k 2024-09-17T23:28:49.791 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 300 k 2024-09-17T23:28:49.791 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 99 k 2024-09-17T23:28:49.791 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 85 k 2024-09-17T23:28:49.791 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.1 M 2024-09-17T23:28:49.791 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 170 k 2024-09-17T23:28:49.791 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrading: 2024-09-17T23:28:49.791 INFO:teuthology.orchestra.run.smithi096.stdout: librados2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.4 M 2024-09-17T23:28:49.791 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.2 M 2024-09-17T23:28:49.791 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-17T23:28:49.791 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T23:28:49.791 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T23:28:49.792 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 21 M 2024-09-17T23:28:49.792 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 32 k 2024-09-17T23:28:49.792 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 2.4 M 2024-09-17T23:28:49.792 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 249 k 2024-09-17T23:28:49.792 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 4.7 M 2024-09-17T23:28:49.792 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 17 M 2024-09-17T23:28:49.792 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 16 k 2024-09-17T23:28:49.792 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 25 k 2024-09-17T23:28:49.792 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T23:28:49.792 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T23:28:49.792 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T23:28:49.792 INFO:teuthology.orchestra.run.smithi096.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T23:28:49.793 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T23:28:49.793 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T23:28:49.793 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T23:28:49.793 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T23:28:49.793 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 163 k 2024-09-17T23:28:49.793 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T23:28:49.793 INFO:teuthology.orchestra.run.smithi096.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T23:28:49.793 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T23:28:49.793 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T23:28:49.793 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T23:28:49.793 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 506 k 2024-09-17T23:28:49.793 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T23:28:49.794 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 5.3 M 2024-09-17T23:28:49.794 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T23:28:49.794 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T23:28:49.794 INFO:teuthology.orchestra.run.smithi096.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T23:28:49.794 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T23:28:49.794 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T23:28:49.794 INFO:teuthology.orchestra.run.smithi096.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T23:28:49.794 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T23:28:49.794 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T23:28:49.794 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T23:28:49.794 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T23:28:49.794 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T23:28:49.794 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T23:28:49.795 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T23:28:49.795 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T23:28:49.795 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 45 k 2024-09-17T23:28:49.795 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 136 k 2024-09-17T23:28:49.795 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T23:28:49.795 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T23:28:49.795 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T23:28:49.795 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T23:28:49.795 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T23:28:49.795 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T23:28:49.795 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T23:28:49.795 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T23:28:49.795 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T23:28:49.795 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T23:28:49.796 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T23:28:49.796 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-17T23:28:49.796 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T23:28:49.796 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T23:28:49.796 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T23:28:49.796 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T23:28:49.796 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T23:28:49.796 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T23:28:49.796 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T23:28:49.796 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T23:28:49.796 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T23:28:49.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T23:28:49.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T23:28:49.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T23:28:49.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T23:28:49.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T23:28:49.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T23:28:49.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T23:28:49.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T23:28:49.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T23:28:49.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T23:28:49.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T23:28:49.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T23:28:49.797 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T23:28:49.798 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T23:28:49.798 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T23:28:49.798 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T23:28:49.798 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T23:28:49.798 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T23:28:49.798 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T23:28:49.798 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T23:28:49.798 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T23:28:49.798 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T23:28:49.798 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T23:28:49.798 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T23:28:49.798 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T23:28:49.799 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T23:28:49.799 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T23:28:49.799 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T23:28:49.799 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T23:28:49.799 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T23:28:49.799 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T23:28:49.799 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T23:28:49.799 INFO:teuthology.orchestra.run.smithi096.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T23:28:49.799 INFO:teuthology.orchestra.run.smithi096.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T23:28:49.799 INFO:teuthology.orchestra.run.smithi096.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T23:28:49.799 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T23:28:49.799 INFO:teuthology.orchestra.run.smithi096.stdout:Installing weak dependencies: 2024-09-17T23:28:49.800 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T23:28:49.800 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:28:49.800 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-17T23:28:49.800 INFO:teuthology.orchestra.run.smithi096.stdout:====================================================================================== 2024-09-17T23:28:49.800 INFO:teuthology.orchestra.run.smithi096.stdout:Install 125 Packages 2024-09-17T23:28:49.800 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrade 2 Packages 2024-09-17T23:28:49.800 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:28:49.800 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 202 M 2024-09-17T23:28:49.801 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-17T23:28:50.200 INFO:teuthology.orchestra.run.smithi096.stdout:(1/127): ceph-19.1.1-330.gc2ddf972.el9.x86_64.r 47 kB/s | 6.5 kB 00:00 2024-09-17T23:28:50.517 INFO:teuthology.orchestra.run.smithi096.stdout:(2/127): ceph-base-19.1.1-330.gc2ddf972.el9.x86 12 MB/s | 5.4 MB 00:00 2024-09-17T23:28:50.550 INFO:teuthology.orchestra.run.smithi096.stdout:(3/127): ceph-fuse-19.1.1-330.gc2ddf972.el9.x86 2.5 MB/s | 897 kB 00:00 2024-09-17T23:28:50.575 INFO:teuthology.orchestra.run.smithi096.stdout:(4/127): ceph-immutable-object-cache-19.1.1-330 2.4 MB/s | 145 kB 00:00 2024-09-17T23:28:50.602 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-09-17T23:28:50.667 INFO:teuthology.orchestra.run.smithi096.stdout:(5/127): ceph-mgr-19.1.1-330.gc2ddf972.el9.x86_ 17 MB/s | 1.5 MB 00:00 2024-09-17T23:28:50.934 INFO:teuthology.orchestra.run.smithi096.stdout:(6/127): ceph-mon-19.1.1-330.gc2ddf972.el9.x86_ 18 MB/s | 4.7 MB 00:00 2024-09-17T23:28:50.993 INFO:teuthology.orchestra.run.smithi096.stdout:(7/127): ceph-mds-19.1.1-330.gc2ddf972.el9.x86_ 5.4 MB/s | 2.4 MB 00:00 2024-09-17T23:28:51.268 INFO:teuthology.orchestra.run.smithi096.stdout:(8/127): ceph-common-19.1.1-330.gc2ddf972.el9.x 18 MB/s | 21 MB 00:01 2024-09-17T23:28:51.302 INFO:teuthology.orchestra.run.smithi096.stdout:(9/127): ceph-selinux-19.1.1-330.gc2ddf972.el9. 760 kB/s | 25 kB 00:00 2024-09-17T23:28:51.952 INFO:teuthology.orchestra.run.smithi096.stdout:(10/127): ceph-osd-19.1.1-330.gc2ddf972.el9.x86 17 MB/s | 17 MB 00:01 2024-09-17T23:28:51.994 INFO:teuthology.orchestra.run.smithi096.stdout:(11/127): libcephfs-devel-19.1.1-330.gc2ddf972. 767 kB/s | 32 kB 00:00 2024-09-17T23:28:52.103 INFO:teuthology.orchestra.run.smithi096.stdout:(12/127): libcephfs2-19.1.1-330.gc2ddf972.el9.x 6.8 MB/s | 751 kB 00:00 2024-09-17T23:28:52.161 INFO:teuthology.orchestra.run.smithi096.stdout:(13/127): libcephsqlite-19.1.1-330.gc2ddf972.el 2.7 MB/s | 163 kB 00:00 2024-09-17T23:28:52.228 INFO:teuthology.orchestra.run.smithi096.stdout:(14/127): librados-devel-19.1.1-330.gc2ddf972.e 1.9 MB/s | 127 kB 00:00 2024-09-17T23:28:52.295 INFO:teuthology.orchestra.run.smithi096.stdout:(15/127): libradosstriper1-19.1.1-330.gc2ddf972 7.4 MB/s | 506 kB 00:00 2024-09-17T23:28:52.538 INFO:teuthology.orchestra.run.smithi096.stdout:(16/127): ceph-radosgw-19.1.1-330.gc2ddf972.el9 6.9 MB/s | 11 MB 00:01 2024-09-17T23:28:52.604 INFO:teuthology.orchestra.run.smithi096.stdout:(17/127): python3-ceph-argparse-19.1.1-330.gc2d 693 kB/s | 45 kB 00:00 2024-09-17T23:28:52.688 INFO:teuthology.orchestra.run.smithi096.stdout:(18/127): librgw2-19.1.1-330.gc2ddf972.el9.x86_ 14 MB/s | 5.3 MB 00:00 2024-09-17T23:28:52.713 INFO:teuthology.orchestra.run.smithi096.stdout:(19/127): python3-ceph-common-19.1.1-330.gc2ddf 1.2 MB/s | 136 kB 00:00 2024-09-17T23:28:52.738 INFO:teuthology.orchestra.run.smithi096.stdout:(20/127): python3-cephfs-19.1.1-330.gc2ddf972.e 3.1 MB/s | 162 kB 00:00 2024-09-17T23:28:52.763 INFO:teuthology.orchestra.run.smithi096.stdout:(21/127): python3-rados-19.1.1-330.gc2ddf972.el 6.2 MB/s | 320 kB 00:00 2024-09-17T23:28:52.789 INFO:teuthology.orchestra.run.smithi096.stdout:(22/127): python3-rbd-19.1.1-330.gc2ddf972.el9. 5.8 MB/s | 300 kB 00:00 2024-09-17T23:28:52.804 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-17T23:28:52.805 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-17T23:28:52.814 INFO:teuthology.orchestra.run.smithi096.stdout:(23/127): python3-rgw-19.1.1-330.gc2ddf972.el9. 1.9 MB/s | 99 kB 00:00 2024-09-17T23:28:52.840 INFO:teuthology.orchestra.run.smithi096.stdout:(24/127): rbd-fuse-19.1.1-330.gc2ddf972.el9.x86 1.6 MB/s | 85 kB 00:00 2024-09-17T23:28:52.865 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-09-17T23:28:52.884 INFO:teuthology.orchestra.run.smithi192.stdout:====================================================================================== 2024-09-17T23:28:52.884 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-09-17T23:28:52.884 INFO:teuthology.orchestra.run.smithi192.stdout:====================================================================================== 2024-09-17T23:28:52.884 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-09-17T23:28:52.885 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 6.5 k 2024-09-17T23:28:52.885 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 5.4 M 2024-09-17T23:28:52.885 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 897 k 2024-09-17T23:28:52.885 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 145 k 2024-09-17T23:28:52.885 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 1.5 M 2024-09-17T23:28:52.885 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 146 k 2024-09-17T23:28:52.885 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 3.6 M 2024-09-17T23:28:52.885 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 7.4 M 2024-09-17T23:28:52.885 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 49 k 2024-09-17T23:28:52.885 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 11 M 2024-09-17T23:28:52.885 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 49 M 2024-09-17T23:28:52.885 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 285 k 2024-09-17T23:28:52.886 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 765 k 2024-09-17T23:28:52.886 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 32 k 2024-09-17T23:28:52.886 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 751 k 2024-09-17T23:28:52.886 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 127 k 2024-09-17T23:28:52.886 INFO:teuthology.orchestra.run.smithi192.stdout: pv x86_64 1.6.20-1.el9 epel 65 k 2024-09-17T23:28:52.886 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 162 k 2024-09-17T23:28:52.886 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 320 k 2024-09-17T23:28:52.886 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 300 k 2024-09-17T23:28:52.886 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 99 k 2024-09-17T23:28:52.886 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 85 k 2024-09-17T23:28:52.886 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.1 M 2024-09-17T23:28:52.887 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 170 k 2024-09-17T23:28:52.887 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2024-09-17T23:28:52.887 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.4 M 2024-09-17T23:28:52.887 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 3.2 M 2024-09-17T23:28:52.887 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-09-17T23:28:52.887 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-17T23:28:52.887 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-17T23:28:52.887 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 21 M 2024-09-17T23:28:52.887 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 32 k 2024-09-17T23:28:52.887 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 2.4 M 2024-09-17T23:28:52.888 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 249 k 2024-09-17T23:28:52.888 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 4.7 M 2024-09-17T23:28:52.888 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 17 M 2024-09-17T23:28:52.888 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts noarch 2:19.1.1-330.gc2ddf972.el9 ceph-noarch 16 k 2024-09-17T23:28:52.888 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 25 k 2024-09-17T23:28:52.888 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-17T23:28:52.888 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-17T23:28:52.888 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-17T23:28:52.888 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-17T23:28:52.888 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-17T23:28:52.888 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-17T23:28:52.888 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-17T23:28:52.889 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-17T23:28:52.889 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 163 k 2024-09-17T23:28:52.889 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-17T23:28:52.889 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-17T23:28:52.889 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-17T23:28:52.889 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-17T23:28:52.889 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-17T23:28:52.889 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 506 k 2024-09-17T23:28:52.889 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-17T23:28:52.889 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 5.3 M 2024-09-17T23:28:52.890 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-17T23:28:52.890 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-17T23:28:52.890 INFO:teuthology.orchestra.run.smithi192.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-17T23:28:52.890 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-17T23:28:52.890 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-17T23:28:52.890 INFO:teuthology.orchestra.run.smithi192.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-17T23:28:52.890 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-17T23:28:52.890 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-17T23:28:52.890 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-17T23:28:52.890 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-17T23:28:52.890 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-17T23:28:52.890 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-17T23:28:52.891 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-17T23:28:52.891 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-17T23:28:52.891 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 45 k 2024-09-17T23:28:52.891 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:19.1.1-330.gc2ddf972.el9 ceph 136 k 2024-09-17T23:28:52.891 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-17T23:28:52.891 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-17T23:28:52.891 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-17T23:28:52.891 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-17T23:28:52.891 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-17T23:28:52.891 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-17T23:28:52.891 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-17T23:28:52.891 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-17T23:28:52.891 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-17T23:28:52.891 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-17T23:28:52.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-17T23:28:52.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-17T23:28:52.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-17T23:28:52.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-17T23:28:52.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-17T23:28:52.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-17T23:28:52.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-17T23:28:52.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-17T23:28:52.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-17T23:28:52.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-17T23:28:52.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-17T23:28:52.892 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-17T23:28:52.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-17T23:28:52.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-17T23:28:52.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-17T23:28:52.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-17T23:28:52.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-17T23:28:52.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-17T23:28:52.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-17T23:28:52.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-17T23:28:52.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-17T23:28:52.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-17T23:28:52.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-17T23:28:52.893 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-17T23:28:52.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-17T23:28:52.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-17T23:28:52.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-17T23:28:52.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-17T23:28:52.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-17T23:28:52.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-17T23:28:52.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-17T23:28:52.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-17T23:28:52.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-17T23:28:52.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-17T23:28:52.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-17T23:28:52.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-17T23:28:52.894 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-17T23:28:52.895 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-17T23:28:52.895 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-17T23:28:52.895 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-17T23:28:52.895 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-17T23:28:52.895 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-17T23:28:52.895 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-17T23:28:52.895 INFO:teuthology.orchestra.run.smithi192.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-17T23:28:52.895 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-17T23:28:52.895 INFO:teuthology.orchestra.run.smithi192.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-17T23:28:52.895 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-17T23:28:52.895 INFO:teuthology.orchestra.run.smithi192.stdout:Installing weak dependencies: 2024-09-17T23:28:52.895 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-17T23:28:52.895 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:28:52.895 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-09-17T23:28:52.896 INFO:teuthology.orchestra.run.smithi192.stdout:====================================================================================== 2024-09-17T23:28:52.896 INFO:teuthology.orchestra.run.smithi192.stdout:Install 125 Packages 2024-09-17T23:28:52.896 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 2 Packages 2024-09-17T23:28:52.896 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:28:52.896 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 202 M 2024-09-17T23:28:52.896 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-09-17T23:28:52.898 INFO:teuthology.orchestra.run.smithi096.stdout:(25/127): rbd-nbd-19.1.1-330.gc2ddf972.el9.x86_ 2.9 MB/s | 170 kB 00:00 2024-09-17T23:28:52.966 INFO:teuthology.orchestra.run.smithi096.stdout:(26/127): ceph-grafana-dashboards-19.1.1-330.gc 479 kB/s | 32 kB 00:00 2024-09-17T23:28:53.383 INFO:teuthology.orchestra.run.smithi096.stdout:(27/127): ceph-test-19.1.1-330.gc2ddf972.el9.x8 23 MB/s | 49 MB 00:02 2024-09-17T23:28:53.408 INFO:teuthology.orchestra.run.smithi096.stdout:(28/127): ceph-mgr-cephadm-19.1.1-330.gc2ddf972 331 kB/s | 146 kB 00:00 2024-09-17T23:28:53.467 INFO:teuthology.orchestra.run.smithi096.stdout:(29/127): rbd-mirror-19.1.1-330.gc2ddf972.el9.x 4.8 MB/s | 3.1 MB 00:00 2024-09-17T23:28:53.559 INFO:teuthology.orchestra.run.smithi096.stdout:(30/127): ceph-mgr-dashboard-19.1.1-330.gc2ddf9 20 MB/s | 3.6 MB 00:00 2024-09-17T23:28:53.584 INFO:teuthology.orchestra.run.smithi096.stdout:(31/127): ceph-mgr-modules-core-19.1.1-330.gc2d 2.1 MB/s | 249 kB 00:00 2024-09-17T23:28:53.609 INFO:teuthology.orchestra.run.smithi096.stdout:(32/127): ceph-mgr-rook-19.1.1-330.gc2ddf972.el 986 kB/s | 49 kB 00:00 2024-09-17T23:28:53.634 INFO:teuthology.orchestra.run.smithi096.stdout:(33/127): ceph-prometheus-alerts-19.1.1-330.gc2 328 kB/s | 16 kB 00:00 2024-09-17T23:28:53.659 INFO:teuthology.orchestra.run.smithi096.stdout:(34/127): ceph-volume-19.1.1-330.gc2ddf972.el9. 5.6 MB/s | 285 kB 00:00 2024-09-17T23:28:53.735 INFO:teuthology.orchestra.run.smithi096.stdout:(35/127): cephadm-19.1.1-330.gc2ddf972.el9.noar 7.5 MB/s | 765 kB 00:00 2024-09-17T23:28:53.818 INFO:teuthology.orchestra.run.smithi096.stdout:(36/127): ceph-mgr-diskprediction-local-19.1.1- 18 MB/s | 7.4 MB 00:00 2024-09-17T23:28:53.838 INFO:teuthology.orchestra.run.smithi192.stdout:(1/127): ceph-19.1.1-330.gc2ddf972.el9.x86_64.r 52 kB/s | 6.5 kB 00:00 2024-09-17T23:28:53.844 INFO:teuthology.orchestra.run.smithi096.stdout:(37/127): fuse-2.9.9-16.el9.x86_64.rpm 434 kB/s | 80 kB 00:00 2024-09-17T23:28:53.985 INFO:teuthology.orchestra.run.smithi096.stdout:(38/127): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-09-17T23:28:54.011 INFO:teuthology.orchestra.run.smithi096.stdout:(39/127): libquadmath-11.5.0-2.el9.x86_64.rpm 995 kB/s | 191 kB 00:00 2024-09-17T23:28:54.061 INFO:teuthology.orchestra.run.smithi096.stdout:(40/127): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-17T23:28:54.079 INFO:teuthology.orchestra.run.smithi192.stdout:(2/127): ceph-fuse-19.1.1-330.gc2ddf972.el9.x86 3.6 MB/s | 897 kB 00:00 2024-09-17T23:28:54.120 INFO:teuthology.orchestra.run.smithi096.stdout:(41/127): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-09-17T23:28:54.145 INFO:teuthology.orchestra.run.smithi096.stdout:(42/127): libgfortran-11.5.0-2.el9.x86_64.rpm 1.9 MB/s | 801 kB 00:00 2024-09-17T23:28:54.154 INFO:teuthology.orchestra.run.smithi192.stdout:(3/127): ceph-immutable-object-cache-19.1.1-330 1.9 MB/s | 145 kB 00:00 2024-09-17T23:28:54.170 INFO:teuthology.orchestra.run.smithi096.stdout:(43/127): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-09-17T23:28:54.229 INFO:teuthology.orchestra.run.smithi096.stdout:(44/127): python3-urllib3-1.26.5-6.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-09-17T23:28:54.255 INFO:teuthology.orchestra.run.smithi192.stdout:(4/127): ceph-base-19.1.1-330.gc2ddf972.el9.x86 9.9 MB/s | 5.4 MB 00:00 2024-09-17T23:28:54.346 INFO:teuthology.orchestra.run.smithi096.stdout:(45/127): python3-cryptography-36.0.1-4.el9.x86 3.4 MB/s | 1.2 MB 00:00 2024-09-17T23:28:54.363 INFO:teuthology.orchestra.run.smithi096.stdout:(46/127): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 33 kB 00:00 2024-09-17T23:28:54.363 INFO:teuthology.orchestra.run.smithi192.stdout:(5/127): ceph-mgr-19.1.1-330.gc2ddf972.el9.x86_ 14 MB/s | 1.5 MB 00:00 2024-09-17T23:28:54.388 INFO:teuthology.orchestra.run.smithi096.stdout:(47/127): boost-program-options-1.75.0-8.el9.x8 491 kB/s | 107 kB 00:00 2024-09-17T23:28:54.405 INFO:teuthology.orchestra.run.smithi096.stdout:(48/127): flexiblas-openblas-openmp-3.0.4-8.el9 433 kB/s | 18 kB 00:00 2024-09-17T23:28:54.447 INFO:teuthology.orchestra.run.smithi096.stdout:(49/127): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-17T23:28:54.531 INFO:teuthology.orchestra.run.smithi096.stdout:(50/127): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 160 kB 00:00 2024-09-17T23:28:54.631 INFO:teuthology.orchestra.run.smithi096.stdout:(51/127): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 662 kB 00:00 2024-09-17T23:28:54.657 INFO:teuthology.orchestra.run.smithi096.stdout:(52/127): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-17T23:28:54.690 INFO:teuthology.orchestra.run.smithi096.stdout:(53/127): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-09-17T23:28:54.715 INFO:teuthology.orchestra.run.smithi096.stdout:(54/127): openblas-0.3.26-2.el9.x86_64.rpm 658 kB/s | 38 kB 00:00 2024-09-17T23:28:54.730 INFO:teuthology.orchestra.run.smithi192.stdout:(6/127): ceph-common-19.1.1-330.gc2ddf972.el9.x 21 MB/s | 21 MB 00:01 2024-09-17T23:28:54.806 INFO:teuthology.orchestra.run.smithi192.stdout:(7/127): ceph-mon-19.1.1-330.gc2ddf972.el9.x86_ 11 MB/s | 4.7 MB 00:00 2024-09-17T23:28:54.841 INFO:teuthology.orchestra.run.smithi096.stdout:(55/127): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-09-17T23:28:54.931 INFO:teuthology.orchestra.run.smithi192.stdout:(8/127): ceph-mds-19.1.1-330.gc2ddf972.el9.x86_ 3.1 MB/s | 2.4 MB 00:00 2024-09-17T23:28:55.039 INFO:teuthology.orchestra.run.smithi192.stdout:(9/127): ceph-selinux-19.1.1-330.gc2ddf972.el9. 232 kB/s | 25 kB 00:00 2024-09-17T23:28:55.431 INFO:teuthology.orchestra.run.smithi192.stdout:(10/127): ceph-osd-19.1.1-330.gc2ddf972.el9.x86 24 MB/s | 17 MB 00:00 2024-09-17T23:28:55.449 INFO:teuthology.orchestra.run.smithi096.stdout:(56/127): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.7 MB/s | 3.0 MB 00:01 2024-09-17T23:28:55.548 INFO:teuthology.orchestra.run.smithi192.stdout:(11/127): ceph-radosgw-19.1.1-330.gc2ddf972.el9 14 MB/s | 11 MB 00:00 2024-09-17T23:28:55.573 INFO:teuthology.orchestra.run.smithi192.stdout:(12/127): libcephfs-devel-19.1.1-330.gc2ddf972. 224 kB/s | 32 kB 00:00 2024-09-17T23:28:55.600 INFO:teuthology.orchestra.run.smithi096.stdout:(57/127): python3-devel-3.9.19-8.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-09-17T23:28:55.615 INFO:teuthology.orchestra.run.smithi192.stdout:(13/127): libcephfs2-19.1.1-330.gc2ddf972.el9.x 11 MB/s | 751 kB 00:00 2024-09-17T23:28:55.641 INFO:teuthology.orchestra.run.smithi192.stdout:(14/127): libcephsqlite-19.1.1-330.gc2ddf972.el 2.4 MB/s | 163 kB 00:00 2024-09-17T23:28:55.666 INFO:teuthology.orchestra.run.smithi192.stdout:(15/127): librados-devel-19.1.1-330.gc2ddf972.e 2.5 MB/s | 127 kB 00:00 2024-09-17T23:28:55.699 INFO:teuthology.orchestra.run.smithi192.stdout:(16/127): libradosstriper1-19.1.1-330.gc2ddf972 8.4 MB/s | 506 kB 00:00 2024-09-17T23:28:55.742 INFO:teuthology.orchestra.run.smithi096.stdout:(58/127): python3-jinja2-2.11.3-6.el9.noarch.rp 1.7 MB/s | 248 kB 00:00 2024-09-17T23:28:55.758 INFO:teuthology.orchestra.run.smithi192.stdout:(17/127): python3-ceph-argparse-19.1.1-330.gc2d 774 kB/s | 45 kB 00:00 2024-09-17T23:28:55.792 INFO:teuthology.orchestra.run.smithi096.stdout:(59/127): python3-jsonpatch-1.21-16.el9.noarch. 512 kB/s | 26 kB 00:00 2024-09-17T23:28:55.817 INFO:teuthology.orchestra.run.smithi192.stdout:(18/127): python3-ceph-common-19.1.1-330.gc2ddf 2.3 MB/s | 136 kB 00:00 2024-09-17T23:28:55.842 INFO:teuthology.orchestra.run.smithi096.stdout:(60/127): python3-jsonpointer-2.0-4.el9.noarch. 384 kB/s | 19 kB 00:00 2024-09-17T23:28:55.892 INFO:teuthology.orchestra.run.smithi192.stdout:(19/127): librgw2-19.1.1-330.gc2ddf972.el9.x86_ 24 MB/s | 5.3 MB 00:00 2024-09-17T23:28:55.917 INFO:teuthology.orchestra.run.smithi192.stdout:(20/127): python3-cephfs-19.1.1-330.gc2ddf972.e 1.6 MB/s | 162 kB 00:00 2024-09-17T23:28:55.943 INFO:teuthology.orchestra.run.smithi192.stdout:(21/127): python3-rados-19.1.1-330.gc2ddf972.el 6.2 MB/s | 320 kB 00:00 2024-09-17T23:28:55.951 INFO:teuthology.orchestra.run.smithi096.stdout:(61/127): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-09-17T23:28:55.977 INFO:teuthology.orchestra.run.smithi192.stdout:(22/127): python3-rbd-19.1.1-330.gc2ddf972.el9. 5.0 MB/s | 300 kB 00:00 2024-09-17T23:28:56.001 INFO:teuthology.orchestra.run.smithi096.stdout:(62/127): python3-markupsafe-1.1.1-12.el9.x86_6 696 kB/s | 35 kB 00:00 2024-09-17T23:28:56.004 INFO:teuthology.orchestra.run.smithi192.stdout:(23/127): python3-rgw-19.1.1-330.gc2ddf972.el9. 1.7 MB/s | 99 kB 00:00 2024-09-17T23:28:56.027 INFO:teuthology.orchestra.run.smithi192.stdout:(24/127): rbd-fuse-19.1.1-330.gc2ddf972.el9.x86 1.7 MB/s | 85 kB 00:00 2024-09-17T23:28:56.094 INFO:teuthology.orchestra.run.smithi192.stdout:(25/127): rbd-nbd-19.1.1-330.gc2ddf972.el9.x86_ 2.5 MB/s | 170 kB 00:00 2024-09-17T23:28:56.178 INFO:teuthology.orchestra.run.smithi192.stdout:(26/127): rbd-mirror-19.1.1-330.gc2ddf972.el9.x 18 MB/s | 3.1 MB 00:00 2024-09-17T23:28:56.203 INFO:teuthology.orchestra.run.smithi192.stdout:(27/127): ceph-grafana-dashboards-19.1.1-330.gc 293 kB/s | 32 kB 00:00 2024-09-17T23:28:56.228 INFO:teuthology.orchestra.run.smithi192.stdout:(28/127): ceph-mgr-cephadm-19.1.1-330.gc2ddf972 2.8 MB/s | 146 kB 00:00 2024-09-17T23:28:56.379 INFO:teuthology.orchestra.run.smithi192.stdout:(29/127): ceph-mgr-dashboard-19.1.1-330.gc2ddf9 20 MB/s | 3.6 MB 00:00 2024-09-17T23:28:56.454 INFO:teuthology.orchestra.run.smithi192.stdout:(30/127): ceph-mgr-modules-core-19.1.1-330.gc2d 3.3 MB/s | 249 kB 00:00 2024-09-17T23:28:56.521 INFO:teuthology.orchestra.run.smithi192.stdout:(31/127): ceph-mgr-rook-19.1.1-330.gc2ddf972.el 743 kB/s | 49 kB 00:00 2024-09-17T23:28:56.605 INFO:teuthology.orchestra.run.smithi192.stdout:(32/127): ceph-mgr-diskprediction-local-19.1.1- 20 MB/s | 7.4 MB 00:00 2024-09-17T23:28:56.630 INFO:teuthology.orchestra.run.smithi192.stdout:(33/127): ceph-prometheus-alerts-19.1.1-330.gc2 152 kB/s | 16 kB 00:00 2024-09-17T23:28:56.655 INFO:teuthology.orchestra.run.smithi192.stdout:(34/127): ceph-volume-19.1.1-330.gc2ddf972.el9. 5.6 MB/s | 285 kB 00:00 2024-09-17T23:28:56.697 INFO:teuthology.orchestra.run.smithi192.stdout:(35/127): cephadm-19.1.1-330.gc2ddf972.el9.noar 11 MB/s | 765 kB 00:00 2024-09-17T23:28:56.868 INFO:teuthology.orchestra.run.smithi096.stdout:(63/127): openblas-openmp-0.3.26-2.el9.x86_64.r 2.3 MB/s | 4.9 MB 00:02 2024-09-17T23:28:57.010 INFO:teuthology.orchestra.run.smithi096.stdout:(64/127): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 250 kB 00:00 2024-09-17T23:28:57.152 INFO:teuthology.orchestra.run.smithi096.stdout:(65/127): python3-oauthlib-3.1.1-5.el9.noarch.r 1.5 MB/s | 222 kB 00:00 2024-09-17T23:28:57.235 INFO:teuthology.orchestra.run.smithi096.stdout:(66/127): python3-packaging-20.9-5.el9.noarch.r 926 kB/s | 77 kB 00:00 2024-09-17T23:28:57.286 INFO:teuthology.orchestra.run.smithi096.stdout:(67/127): python3-prettytable-0.7.2-27.el9.noar 845 kB/s | 42 kB 00:00 2024-09-17T23:28:57.347 INFO:teuthology.orchestra.run.smithi192.stdout:(36/127): fuse-2.9.9-16.el9.x86_64.rpm 115 kB/s | 80 kB 00:00 2024-09-17T23:28:57.453 INFO:teuthology.orchestra.run.smithi096.stdout:(68/127): python3-protobuf-3.14.0-13.el9.noarch 1.6 MB/s | 269 kB 00:00 2024-09-17T23:28:57.570 INFO:teuthology.orchestra.run.smithi096.stdout:(69/127): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-17T23:28:57.745 INFO:teuthology.orchestra.run.smithi096.stdout:(70/127): python3-pyasn1-modules-0.4.8-6.el9.no 1.6 MB/s | 279 kB 00:00 2024-09-17T23:28:57.800 INFO:teuthology.orchestra.run.smithi096.stdout:(71/127): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-17T23:28:57.864 INFO:teuthology.orchestra.run.smithi192.stdout:(37/127): ceph-test-19.1.1-330.gc2ddf972.el9.x8 17 MB/s | 49 MB 00:02 2024-09-17T23:28:57.870 INFO:teuthology.orchestra.run.smithi096.stdout:(72/127): python3-babel-2.9.1-2.el9.noarch.rpm 2.0 MB/s | 6.0 MB 00:03 2024-09-17T23:28:57.896 INFO:teuthology.orchestra.run.smithi096.stdout:(73/127): python3-requests-oauthlib-1.3.0-12.el 534 kB/s | 54 kB 00:00 2024-09-17T23:28:57.946 INFO:teuthology.orchestra.run.smithi096.stdout:(74/127): python3-toml-0.10.2-6.el9.noarch.rpm 837 kB/s | 42 kB 00:00 2024-09-17T23:28:57.981 INFO:teuthology.orchestra.run.smithi192.stdout:(38/127): libgfortran-11.5.0-2.el9.x86_64.rpm 624 kB/s | 801 kB 00:01 2024-09-17T23:28:58.096 INFO:teuthology.orchestra.run.smithi096.stdout:(75/127): qatlib-24.02.0-1.el9.x86_64.rpm 1.4 MB/s | 221 kB 00:00 2024-09-17T23:28:58.147 INFO:teuthology.orchestra.run.smithi096.stdout:(76/127): qatlib-service-24.02.0-1.el9.x86_64.r 724 kB/s | 36 kB 00:00 2024-09-17T23:28:58.197 INFO:teuthology.orchestra.run.smithi096.stdout:(77/127): qatzip-libs-1.2.0-1.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-09-17T23:28:58.372 INFO:teuthology.orchestra.run.smithi096.stdout:(78/127): socat-1.7.4.1-6.el9.x86_64.rpm 1.7 MB/s | 304 kB 00:00 2024-09-17T23:28:58.456 INFO:teuthology.orchestra.run.smithi096.stdout:(79/127): xmlstarlet-1.6.1-20.el9.x86_64.rpm 765 kB/s | 64 kB 00:00 2024-09-17T23:28:58.521 INFO:teuthology.orchestra.run.smithi192.stdout:(39/127): libquadmath-11.5.0-2.el9.x86_64.rpm 172 kB/s | 191 kB 00:01 2024-09-17T23:28:58.632 INFO:teuthology.orchestra.run.smithi192.stdout:(40/127): python3-cffi-1.14.5-5.el9.x86_64.rpm 330 kB/s | 253 kB 00:00 2024-09-17T23:28:58.673 INFO:teuthology.orchestra.run.smithi096.stdout:(80/127): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.9 MB/s | 5.1 MB 00:02 2024-09-17T23:28:58.915 INFO:teuthology.orchestra.run.smithi192.stdout:(41/127): python3-cryptography-36.0.1-4.el9.x86 1.3 MB/s | 1.2 MB 00:00 2024-09-17T23:28:58.990 INFO:teuthology.orchestra.run.smithi096.stdout:(81/127): lua-devel-5.4.4-4.el9.x86_64.rpm 42 kB/s | 22 kB 00:00 2024-09-17T23:28:59.124 INFO:teuthology.orchestra.run.smithi096.stdout:(82/127): abseil-cpp-20211102.0-3.el9.x86_64.rp 4.1 MB/s | 548 kB 00:00 2024-09-17T23:28:59.182 INFO:teuthology.orchestra.run.smithi192.stdout:(42/127): python3-ply-3.11-14.el9.noarch.rpm 147 kB/s | 106 kB 00:00 2024-09-17T23:28:59.190 INFO:teuthology.orchestra.run.smithi096.stdout:(83/127): gperftools-libs-2.9.1-3.el9.x86_64.rp 4.5 MB/s | 308 kB 00:00 2024-09-17T23:28:59.207 INFO:teuthology.orchestra.run.smithi096.stdout:(84/127): grpc-data-1.46.7-10.el9.noarch.rpm 1.2 MB/s | 19 kB 00:00 2024-09-17T23:28:59.449 INFO:teuthology.orchestra.run.smithi192.stdout:(43/127): python3-pycparser-2.20-6.el9.noarch.r 165 kB/s | 135 kB 00:00 2024-09-17T23:28:59.633 INFO:teuthology.orchestra.run.smithi192.stdout:(44/127): python3-requests-2.25.1-8.el9.noarch. 174 kB/s | 125 kB 00:00 2024-09-17T23:28:59.849 INFO:teuthology.orchestra.run.smithi096.stdout:(85/127): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.2 MB/s | 16 MB 00:01 2024-09-17T23:28:59.891 INFO:teuthology.orchestra.run.smithi192.stdout:(45/127): boost-program-options-1.75.0-8.el9.x8 242 kB/s | 107 kB 00:00 2024-09-17T23:28:59.916 INFO:teuthology.orchestra.run.smithi192.stdout:(46/127): flexiblas-3.0.4-8.el9.x86_64.rpm 115 kB/s | 33 kB 00:00 2024-09-17T23:28:59.924 INFO:teuthology.orchestra.run.smithi096.stdout:(86/127): libarrow-9.0.0-11.el9.x86_64.rpm 6.2 MB/s | 4.4 MB 00:00 2024-09-17T23:28:59.942 INFO:teuthology.orchestra.run.smithi192.stdout:(47/127): python3-urllib3-1.26.5-6.el9.noarch.r 283 kB/s | 215 kB 00:00 2024-09-17T23:28:59.950 INFO:teuthology.orchestra.run.smithi096.stdout:(87/127): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-17T23:28:59.975 INFO:teuthology.orchestra.run.smithi096.stdout:(88/127): libarrow-doc-9.0.0-11.el9.noarch.rpm 207 kB/s | 26 kB 00:00 2024-09-17T23:29:00.009 INFO:teuthology.orchestra.run.smithi096.stdout:(89/127): protobuf-compiler-3.14.0-13.el9.x86_6 650 kB/s | 863 kB 00:01 2024-09-17T23:29:00.034 INFO:teuthology.orchestra.run.smithi096.stdout:(90/127): libunwind-1.6.2-1.el9.x86_64.rpm 801 kB/s | 67 kB 00:00 2024-09-17T23:29:00.067 INFO:teuthology.orchestra.run.smithi096.stdout:(91/127): luarocks-3.9.2-5.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-09-17T23:29:00.101 INFO:teuthology.orchestra.run.smithi096.stdout:(92/127): pv-1.6.20-1.el9.x86_64.rpm 975 kB/s | 65 kB 00:00 2024-09-17T23:29:00.135 INFO:teuthology.orchestra.run.smithi096.stdout:(93/127): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-09-17T23:29:00.167 INFO:teuthology.orchestra.run.smithi192.stdout:(48/127): flexiblas-openblas-openmp-3.0.4-8.el9 73 kB/s | 18 kB 00:00 2024-09-17T23:29:00.169 INFO:teuthology.orchestra.run.smithi096.stdout:(94/127): python3-cachetools-4.2.4-1.el9.noarch 966 kB/s | 32 kB 00:00 2024-09-17T23:29:00.202 INFO:teuthology.orchestra.run.smithi096.stdout:(95/127): python3-asyncssh-2.13.2-5.el9.noarch. 4.0 MB/s | 548 kB 00:00 2024-09-17T23:29:00.227 INFO:teuthology.orchestra.run.smithi096.stdout:(96/127): python3-certifi-2023.05.07-4.el9.noar 241 kB/s | 14 kB 00:00 2024-09-17T23:29:00.253 INFO:teuthology.orchestra.run.smithi096.stdout:(97/127): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.4 MB/s | 839 kB 00:00 2024-09-17T23:29:00.278 INFO:teuthology.orchestra.run.smithi096.stdout:(98/127): python3-cheroot-8.6.0-4.el9.noarch.rp 2.2 MB/s | 172 kB 00:00 2024-09-17T23:29:00.312 INFO:teuthology.orchestra.run.smithi096.stdout:(99/127): python3-google-auth-2.34.0-1.el9.noar 3.7 MB/s | 220 kB 00:00 2024-09-17T23:29:00.345 INFO:teuthology.orchestra.run.smithi096.stdout:(100/127): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-09-17T23:29:00.371 INFO:teuthology.orchestra.run.smithi096.stdout:(101/127): python3-jaraco-8.2.1-3.el9.noarch.rp 426 kB/s | 11 kB 00:00 2024-09-17T23:29:00.376 INFO:teuthology.orchestra.run.smithi192.stdout:(49/127): libpmemobj-1.12.1-1.el9.x86_64.rpm 370 kB/s | 160 kB 00:00 2024-09-17T23:29:00.396 INFO:teuthology.orchestra.run.smithi096.stdout:(102/127): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-09-17T23:29:00.421 INFO:teuthology.orchestra.run.smithi096.stdout:(103/127): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-09-17T23:29:00.447 INFO:teuthology.orchestra.run.smithi096.stdout:(104/127): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-09-17T23:29:00.472 INFO:teuthology.orchestra.run.smithi096.stdout:(105/127): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-09-17T23:29:00.497 INFO:teuthology.orchestra.run.smithi096.stdout:(106/127): python3-jaraco-text-3.2.0-6.el9.noar 389 kB/s | 20 kB 00:00 2024-09-17T23:29:00.501 INFO:teuthology.orchestra.run.smithi192.stdout:(50/127): librabbitmq-0.11.0-7.el9.x86_64.rpm 135 kB/s | 45 kB 00:00 2024-09-17T23:29:00.531 INFO:teuthology.orchestra.run.smithi096.stdout:(107/127): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-09-17T23:29:00.573 INFO:teuthology.orchestra.run.smithi096.stdout:(108/127): python3-more-itertools-8.12.0-2.el9. 1.8 MB/s | 79 kB 00:00 2024-09-17T23:29:00.607 INFO:teuthology.orchestra.run.smithi096.stdout:(109/127): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-09-17T23:29:00.690 INFO:teuthology.orchestra.run.smithi096.stdout:(110/127): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-09-17T23:29:00.740 INFO:teuthology.orchestra.run.smithi096.stdout:(111/127): python3-grpcio-1.46.7-10.el9.x86_64. 4.4 MB/s | 2.0 MB 00:00 2024-09-17T23:29:00.766 INFO:teuthology.orchestra.run.smithi096.stdout:(112/127): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-09-17T23:29:00.768 INFO:teuthology.orchestra.run.smithi192.stdout:(51/127): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.4 MB/s | 3.0 MB 00:00 2024-09-17T23:29:00.791 INFO:teuthology.orchestra.run.smithi096.stdout:(113/127): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-09-17T23:29:00.816 INFO:teuthology.orchestra.run.smithi096.stdout:(114/127): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-09-17T23:29:00.850 INFO:teuthology.orchestra.run.smithi096.stdout:(115/127): python3-kubernetes-26.1.0-2.el9.noar 2.7 MB/s | 1.0 MB 00:00 2024-09-17T23:29:00.875 INFO:teuthology.orchestra.run.smithi096.stdout:(116/127): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-09-17T23:29:00.901 INFO:teuthology.orchestra.run.smithi096.stdout:(117/127): python3-tempora-5.0.0-2.el9.noarch.r 712 kB/s | 36 kB 00:00 2024-09-17T23:29:00.926 INFO:teuthology.orchestra.run.smithi096.stdout:(118/127): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-09-17T23:29:00.951 INFO:teuthology.orchestra.run.smithi096.stdout:(119/127): python3-typing-extensions-4.12.2-2.e 988 kB/s | 74 kB 00:00 2024-09-17T23:29:00.951 INFO:teuthology.orchestra.run.smithi192.stdout:(52/127): librdkafka-1.6.1-102.el9.x86_64.rpm 1.1 MB/s | 662 kB 00:00 2024-09-17T23:29:00.976 INFO:teuthology.orchestra.run.smithi096.stdout:(120/127): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-17T23:29:01.002 INFO:teuthology.orchestra.run.smithi096.stdout:(121/127): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-09-17T23:29:01.019 INFO:teuthology.orchestra.run.smithi096.stdout:(122/127): python3-zc-lockfile-2.0-10.el9.noarc 478 kB/s | 20 kB 00:00 2024-09-17T23:29:01.035 INFO:teuthology.orchestra.run.smithi192.stdout:(53/127): lttng-ust-2.12.0-6.el9.x86_64.rpm 547 kB/s | 292 kB 00:00 2024-09-17T23:29:01.052 INFO:teuthology.orchestra.run.smithi096.stdout:(123/127): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-17T23:29:01.077 INFO:teuthology.orchestra.run.smithi096.stdout:(124/127): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-09-17T23:29:01.227 INFO:teuthology.orchestra.run.smithi192.stdout:(54/127): openblas-0.3.26-2.el9.x86_64.rpm 140 kB/s | 38 kB 00:00 2024-09-17T23:29:01.253 INFO:teuthology.orchestra.run.smithi096.stdout:(125/127): thrift-0.15.0-2.el9.x86_64.rpm 6.8 MB/s | 1.6 MB 00:00 2024-09-17T23:29:01.269 INFO:teuthology.orchestra.run.smithi192.stdout:(55/127): lua-5.4.4-4.el9.x86_64.rpm 376 kB/s | 188 kB 00:00 2024-09-17T23:29:01.361 INFO:teuthology.orchestra.run.smithi096.stdout:(126/127): librbd1-19.1.1-330.gc2ddf972.el9.x86 11 MB/s | 3.2 MB 00:00 2024-09-17T23:29:01.478 INFO:teuthology.orchestra.run.smithi096.stdout:(127/127): librados2-19.1.1-330.gc2ddf972.el9.x 8.0 MB/s | 3.4 MB 00:00 2024-09-17T23:29:01.484 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:29:01.484 INFO:teuthology.orchestra.run.smithi096.stdout:Total 17 MB/s | 202 MB 00:11 2024-09-17T23:29:01.956 INFO:teuthology.orchestra.run.smithi192.stdout:(56/127): openblas-openmp-0.3.26-2.el9.x86_64.r 5.4 MB/s | 4.9 MB 00:00 2024-09-17T23:29:02.028 INFO:teuthology.orchestra.run.smithi192.stdout:(57/127): protobuf-3.14.0-13.el9.x86_64.rpm 1.3 MB/s | 1.0 MB 00:00 2024-09-17T23:29:02.300 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-17T23:29:02.405 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-17T23:29:02.405 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-17T23:29:02.470 INFO:teuthology.orchestra.run.smithi192.stdout:(58/127): python3-devel-3.9.19-8.el9.x86_64.rpm 474 kB/s | 245 kB 00:00 2024-09-17T23:29:02.620 INFO:teuthology.orchestra.run.smithi192.stdout:(59/127): python3-jinja2-2.11.3-6.el9.noarch.rp 419 kB/s | 248 kB 00:00 2024-09-17T23:29:02.762 INFO:teuthology.orchestra.run.smithi192.stdout:(60/127): python3-jsonpatch-1.21-16.el9.noarch. 88 kB/s | 26 kB 00:00 2024-09-17T23:29:02.887 INFO:teuthology.orchestra.run.smithi192.stdout:(61/127): python3-jsonpointer-2.0-4.el9.noarch. 72 kB/s | 19 kB 00:00 2024-09-17T23:29:02.963 INFO:teuthology.orchestra.run.smithi192.stdout:(62/127): python3-babel-2.9.1-2.el9.noarch.rpm 3.5 MB/s | 6.0 MB 00:01 2024-09-17T23:29:03.155 INFO:teuthology.orchestra.run.smithi192.stdout:(63/127): python3-markupsafe-1.1.1-12.el9.x86_6 130 kB/s | 35 kB 00:00 2024-09-17T23:29:03.213 INFO:teuthology.orchestra.run.smithi192.stdout:(64/127): python3-mako-1.1.4-6.el9.noarch.rpm 382 kB/s | 172 kB 00:00 2024-09-17T23:29:03.656 INFO:teuthology.orchestra.run.smithi192.stdout:(65/127): python3-numpy-f2py-1.20.1-5.el9.x86_6 501 kB/s | 250 kB 00:00 2024-09-17T23:29:03.722 INFO:teuthology.orchestra.run.smithi192.stdout:(66/127): python3-oauthlib-3.1.1-5.el9.noarch.r 437 kB/s | 222 kB 00:00 2024-09-17T23:29:03.906 INFO:teuthology.orchestra.run.smithi192.stdout:(67/127): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.4 MB/s | 5.1 MB 00:00 2024-09-17T23:29:04.073 INFO:teuthology.orchestra.run.smithi192.stdout:(68/127): python3-prettytable-0.7.2-27.el9.noar 121 kB/s | 42 kB 00:00 2024-09-17T23:29:04.098 INFO:teuthology.orchestra.run.smithi192.stdout:(69/127): python3-packaging-20.9-5.el9.noarch.r 174 kB/s | 77 kB 00:00 2024-09-17T23:29:04.311 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-17T23:29:04.312 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-17T23:29:04.498 INFO:teuthology.orchestra.run.smithi192.stdout:(70/127): python3-protobuf-3.14.0-13.el9.noarch 455 kB/s | 269 kB 00:00 2024-09-17T23:29:04.523 INFO:teuthology.orchestra.run.smithi192.stdout:(71/127): python3-pyasn1-0.4.8-6.el9.noarch.rpm 352 kB/s | 159 kB 00:00 2024-09-17T23:29:04.624 INFO:teuthology.orchestra.run.smithi192.stdout:(72/127): python3-pyasn1-modules-0.4.8-6.el9.no 531 kB/s | 279 kB 00:00 2024-09-17T23:29:04.840 INFO:teuthology.orchestra.run.smithi192.stdout:(73/127): python3-pytz-2021.1-5.el9.noarch.rpm 149 kB/s | 51 kB 00:00 2024-09-17T23:29:04.890 INFO:teuthology.orchestra.run.smithi192.stdout:(74/127): python3-requests-oauthlib-1.3.0-12.el 146 kB/s | 54 kB 00:00 2024-09-17T23:29:05.116 INFO:teuthology.orchestra.run.smithi192.stdout:(75/127): python3-toml-0.10.2-6.el9.noarch.rpm 151 kB/s | 42 kB 00:00 2024-09-17T23:29:05.383 INFO:teuthology.orchestra.run.smithi192.stdout:(76/127): qatlib-service-24.02.0-1.el9.x86_64.r 134 kB/s | 36 kB 00:00 2024-09-17T23:29:05.408 INFO:teuthology.orchestra.run.smithi192.stdout:(77/127): qatlib-24.02.0-1.el9.x86_64.rpm 427 kB/s | 221 kB 00:00 2024-09-17T23:29:05.725 INFO:teuthology.orchestra.run.smithi192.stdout:(78/127): qatzip-libs-1.2.0-1.el9.x86_64.rpm 138 kB/s | 47 kB 00:00 2024-09-17T23:29:05.917 INFO:teuthology.orchestra.run.smithi192.stdout:(79/127): socat-1.7.4.1-6.el9.x86_64.rpm 598 kB/s | 304 kB 00:00 2024-09-17T23:29:06.026 INFO:teuthology.orchestra.run.smithi192.stdout:(80/127): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-09-17T23:29:06.068 INFO:teuthology.orchestra.run.smithi192.stdout:(81/127): xmlstarlet-1.6.1-20.el9.x86_64.rpm 186 kB/s | 64 kB 00:00 2024-09-17T23:29:06.218 INFO:teuthology.orchestra.run.smithi192.stdout:(82/127): abseil-cpp-20211102.0-3.el9.x86_64.rp 3.6 MB/s | 548 kB 00:00 2024-09-17T23:29:06.252 INFO:teuthology.orchestra.run.smithi192.stdout:(83/127): protobuf-compiler-3.14.0-13.el9.x86_6 3.7 MB/s | 863 kB 00:00 2024-09-17T23:29:06.302 INFO:teuthology.orchestra.run.smithi192.stdout:(84/127): grpc-data-1.46.7-10.el9.noarch.rpm 390 kB/s | 19 kB 00:00 2024-09-17T23:29:06.335 INFO:teuthology.orchestra.run.smithi192.stdout:(85/127): gperftools-libs-2.9.1-3.el9.x86_64.rp 2.6 MB/s | 308 kB 00:00 2024-09-17T23:29:06.369 INFO:teuthology.orchestra.run.smithi192.stdout:(86/127): libarrow-doc-9.0.0-11.el9.noarch.rpm 780 kB/s | 26 kB 00:00 2024-09-17T23:29:06.403 INFO:teuthology.orchestra.run.smithi192.stdout:(87/127): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-17T23:29:06.436 INFO:teuthology.orchestra.run.smithi192.stdout:(88/127): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-17T23:29:06.488 INFO:teuthology.orchestra.run.smithi192.stdout:(89/127): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-09-17T23:29:06.645 INFO:teuthology.orchestra.run.smithi192.stdout:(90/127): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.2 MB/s | 839 kB 00:00 2024-09-17T23:29:06.679 INFO:teuthology.orchestra.run.smithi192.stdout:(91/127): pv-1.6.20-1.el9.x86_64.rpm 1.9 MB/s | 65 kB 00:00 2024-09-17T23:29:06.788 INFO:teuthology.orchestra.run.smithi192.stdout:(92/127): python3-asyncssh-2.13.2-5.el9.noarch. 5.0 MB/s | 548 kB 00:00 2024-09-17T23:29:06.821 INFO:teuthology.orchestra.run.smithi192.stdout:(93/127): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 MB/s | 43 kB 00:00 2024-09-17T23:29:06.847 INFO:teuthology.orchestra.run.smithi192.stdout:(94/127): python3-cachetools-4.2.4-1.el9.noarch 1.3 MB/s | 32 kB 00:00 2024-09-17T23:29:06.872 INFO:teuthology.orchestra.run.smithi192.stdout:(95/127): python3-certifi-2023.05.07-4.el9.noar 570 kB/s | 14 kB 00:00 2024-09-17T23:29:06.922 INFO:teuthology.orchestra.run.smithi192.stdout:(96/127): python3-cheroot-8.6.0-4.el9.noarch.rp 3.4 MB/s | 172 kB 00:00 2024-09-17T23:29:07.014 INFO:teuthology.orchestra.run.smithi192.stdout:(97/127): python3-cherrypy-18.6.1-2.el9.noarch. 3.8 MB/s | 358 kB 00:00 2024-09-17T23:29:07.082 INFO:teuthology.orchestra.run.smithi192.stdout:(98/127): python3-google-auth-2.34.0-1.el9.noar 3.2 MB/s | 220 kB 00:00 2024-09-17T23:29:07.165 INFO:teuthology.orchestra.run.smithi192.stdout:(99/127): libarrow-9.0.0-11.el9.x86_64.rpm 5.1 MB/s | 4.4 MB 00:00 2024-09-17T23:29:07.207 INFO:teuthology.orchestra.run.smithi192.stdout:(100/127): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-09-17T23:29:07.232 INFO:teuthology.orchestra.run.smithi192.stdout:(101/127): python3-jaraco-8.2.1-3.el9.noarch.rp 431 kB/s | 11 kB 00:00 2024-09-17T23:29:07.266 INFO:teuthology.orchestra.run.smithi192.stdout:(102/127): python3-jaraco-classes-3.2.1-5.el9.n 533 kB/s | 18 kB 00:00 2024-09-17T23:29:07.291 INFO:teuthology.orchestra.run.smithi192.stdout:(103/127): python3-jaraco-collections-3.0.0-8.e 936 kB/s | 23 kB 00:00 2024-09-17T23:29:07.316 INFO:teuthology.orchestra.run.smithi192.stdout:(104/127): python3-jaraco-functools-3.5.0-2.el9 775 kB/s | 19 kB 00:00 2024-09-17T23:29:07.342 INFO:teuthology.orchestra.run.smithi192.stdout:(105/127): python3-jaraco-text-3.2.0-6.el9.noar 789 kB/s | 20 kB 00:00 2024-09-17T23:29:07.400 INFO:teuthology.orchestra.run.smithi192.stdout:(106/127): python3-grpcio-1.46.7-10.el9.x86_64. 6.4 MB/s | 2.0 MB 00:00 2024-09-17T23:29:07.425 INFO:teuthology.orchestra.run.smithi192.stdout:(107/127): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-17T23:29:07.451 INFO:teuthology.orchestra.run.smithi192.stdout:(108/127): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-17T23:29:07.484 INFO:teuthology.orchestra.run.smithi192.stdout:(109/127): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-09-17T23:29:07.510 INFO:teuthology.orchestra.run.smithi192.stdout:(110/127): python3-natsort-7.1.1-5.el9.noarch.r 979 kB/s | 58 kB 00:00 2024-09-17T23:29:07.535 INFO:teuthology.orchestra.run.smithi192.stdout:(111/127): python3-portend-3.1.0-2.el9.noarch.r 664 kB/s | 16 kB 00:00 2024-09-17T23:29:07.569 INFO:teuthology.orchestra.run.smithi192.stdout:(112/127): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-09-17T23:29:07.594 INFO:teuthology.orchestra.run.smithi192.stdout:(113/127): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-09-17T23:29:07.619 INFO:teuthology.orchestra.run.smithi192.stdout:(114/127): python3-repoze-lru-0.7-16.el9.noarch 615 kB/s | 31 kB 00:00 2024-09-17T23:29:07.645 INFO:teuthology.orchestra.run.smithi192.stdout:(115/127): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-17T23:29:07.670 INFO:teuthology.orchestra.run.smithi192.stdout:(116/127): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-09-17T23:29:07.695 INFO:teuthology.orchestra.run.smithi192.stdout:(117/127): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-09-17T23:29:07.720 INFO:teuthology.orchestra.run.smithi192.stdout:(118/127): python3-typing-extensions-4.12.2-2.e 1.5 MB/s | 74 kB 00:00 2024-09-17T23:29:07.745 INFO:teuthology.orchestra.run.smithi192.stdout:(119/127): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-09-17T23:29:07.771 INFO:teuthology.orchestra.run.smithi192.stdout:(120/127): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-17T23:29:07.796 INFO:teuthology.orchestra.run.smithi192.stdout:(121/127): python3-zc-lockfile-2.0-10.el9.noarc 804 kB/s | 20 kB 00:00 2024-09-17T23:29:07.821 INFO:teuthology.orchestra.run.smithi192.stdout:(122/127): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-09-17T23:29:07.847 INFO:teuthology.orchestra.run.smithi192.stdout:(123/127): re2-20211101-20.el9.x86_64.rpm 3.7 MB/s | 191 kB 00:00 2024-09-17T23:29:07.980 INFO:teuthology.orchestra.run.smithi192.stdout:(124/127): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-09-17T23:29:08.139 INFO:teuthology.orchestra.run.smithi192.stdout:(125/127): librados2-19.1.1-330.gc2ddf972.el9.x 12 MB/s | 3.4 MB 00:00 2024-09-17T23:29:08.256 INFO:teuthology.orchestra.run.smithi192.stdout:(126/127): librbd1-19.1.1-330.gc2ddf972.el9.x86 11 MB/s | 3.2 MB 00:00 2024-09-17T23:29:09.348 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-17T23:29:09.430 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/129 2024-09-17T23:29:09.503 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/129 2024-09-17T23:29:09.768 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/129 2024-09-17T23:29:10.110 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/129 2024-09-17T23:29:10.227 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 5/129 2024-09-17T23:29:10.349 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 5/129 2024-09-17T23:29:10.387 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 6/129 2024-09-17T23:29:10.479 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 6/129 2024-09-17T23:29:10.530 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 7/129 2024-09-17T23:29:10.560 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/129 2024-09-17T23:29:10.595 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/129 2024-09-17T23:29:10.623 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 10/129 2024-09-17T23:29:10.727 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 10/129 2024-09-17T23:29:10.749 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_ 11/129 2024-09-17T23:29:10.813 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_ 11/129 2024-09-17T23:29:10.842 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 12/129 2024-09-17T23:29:10.863 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 12/129 2024-09-17T23:29:11.211 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 13/129 2024-09-17T23:29:11.242 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 13/129 2024-09-17T23:29:11.349 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : re2-1:20211101-20.el9.x86_64 14/129 2024-09-17T23:29:11.446 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 15/129 2024-09-17T23:29:11.474 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 16/129 2024-09-17T23:29:11.510 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : liboath-2.6.7-2.el9.x86_64 17/129 2024-09-17T23:29:11.601 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 18/129 2024-09-17T23:29:11.661 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 19/129 2024-09-17T23:29:11.702 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-packaging-20.9-5.el9.noarch 20/129 2024-09-17T23:29:11.745 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 21/129 2024-09-17T23:29:11.778 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 22/129 2024-09-17T23:29:11.806 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-5.4.4-4.el9.x86_64 23/129 2024-09-17T23:29:11.876 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 24/129 2024-09-17T23:29:11.929 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 25/129 2024-09-17T23:29:11.967 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 26/129 2024-09-17T23:29:12.015 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 27/129 2024-09-17T23:29:12.097 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 28/129 2024-09-17T23:29:12.130 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x 29/129 2024-09-17T23:29:12.175 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9 30/129 2024-09-17T23:29:12.202 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 31/129 2024-09-17T23:29:12.275 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 32/129 2024-09-17T23:29:12.334 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : luarocks-3.9.2-5.el9.noarch 33/129 2024-09-17T23:29:12.409 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 34/129 2024-09-17T23:29:12.547 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 35/129 2024-09-17T23:29:12.603 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 36/129 2024-09-17T23:29:12.672 INFO:teuthology.orchestra.run.smithi192.stdout:(127/127): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.0 MB/s | 16 MB 00:08 2024-09-17T23:29:12.678 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-09-17T23:29:12.678 INFO:teuthology.orchestra.run.smithi192.stdout:Total 10 MB/s | 202 MB 00:19 2024-09-17T23:29:13.133 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rsa-4.9-2.el9.noarch 37/129 2024-09-17T23:29:13.232 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 38/129 2024-09-17T23:29:13.261 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 39/129 2024-09-17T23:29:13.304 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 40/129 2024-09-17T23:29:13.341 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 41/129 2024-09-17T23:29:13.375 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 42/129 2024-09-17T23:29:13.439 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 43/129 2024-09-17T23:29:13.488 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-09-17T23:29:13.512 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 44/129 2024-09-17T23:29:13.543 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 45/129 2024-09-17T23:29:13.579 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 46/129 2024-09-17T23:29:13.591 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-09-17T23:29:13.591 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-09-17T23:29:13.642 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 47/129 2024-09-17T23:29:13.687 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 48/129 2024-09-17T23:29:13.729 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 49/129 2024-09-17T23:29:13.868 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 50/129 2024-09-17T23:29:13.907 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 51/129 2024-09-17T23:29:13.948 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 52/129 2024-09-17T23:29:14.050 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 53/129 2024-09-17T23:29:14.082 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 54/129 2024-09-17T23:29:14.133 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 55/129 2024-09-17T23:29:14.172 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 56/129 2024-09-17T23:29:14.197 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 57/129 2024-09-17T23:29:14.293 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 58/129 2024-09-17T23:29:14.424 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 59/129 2024-09-17T23:29:14.469 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 60/129 2024-09-17T23:29:14.518 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 61/129 2024-09-17T23:29:14.553 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 62/129 2024-09-17T23:29:14.583 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 63/129 2024-09-17T23:29:14.617 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 64/129 2024-09-17T23:29:14.660 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 65/129 2024-09-17T23:29:14.709 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 66/129 2024-09-17T23:29:14.744 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 67/129 2024-09-17T23:29:14.831 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 68/129 2024-09-17T23:29:14.874 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 69/129 2024-09-17T23:29:14.905 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 70/129 2024-09-17T23:29:15.009 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 71/129 2024-09-17T23:29:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 72/129 2024-09-17T23:29:15.592 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-09-17T23:29:15.592 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-09-17T23:29:15.834 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 73/129 2024-09-17T23:29:15.879 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 74/129 2024-09-17T23:29:15.913 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 75/129 2024-09-17T23:29:16.012 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 76/129 2024-09-17T23:29:16.047 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 77/129 2024-09-17T23:29:16.234 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-0.3.26-2.el9.x86_64 78/129 2024-09-17T23:29:16.264 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 79/129 2024-09-17T23:29:16.343 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 80/129 2024-09-17T23:29:17.003 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 81/129 2024-09-17T23:29:17.114 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 82/129 2024-09-17T23:29:18.415 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/129 2024-09-17T23:29:18.503 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 84/129 2024-09-17T23:29:18.812 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 85/129 2024-09-17T23:29:18.856 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 86/129 2024-09-17T23:29:18.956 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 86/129 2024-09-17T23:29:18.986 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 87/129 2024-09-17T23:29:19.023 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 88/129 2024-09-17T23:29:19.503 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 89/129 2024-09-17T23:29:19.535 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 90/129 2024-09-17T23:29:19.606 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 90/129 2024-09-17T23:29:19.636 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 91/129 2024-09-17T23:29:20.837 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-09-17T23:29:20.908 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/129 2024-09-17T23:29:20.933 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/129 2024-09-17T23:29:20.991 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/129 2024-09-17T23:29:21.334 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 4/129 2024-09-17T23:29:21.451 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 5/129 2024-09-17T23:29:21.574 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 5/129 2024-09-17T23:29:21.602 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 6/129 2024-09-17T23:29:21.693 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 6/129 2024-09-17T23:29:21.741 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 7/129 2024-09-17T23:29:21.773 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/129 2024-09-17T23:29:21.810 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/129 2024-09-17T23:29:21.838 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 10/129 2024-09-17T23:29:21.943 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 10/129 2024-09-17T23:29:21.966 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 92/129 2024-09-17T23:29:21.972 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_ 11/129 2024-09-17T23:29:22.000 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 92/129 2024-09-17T23:29:22.079 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_ 11/129 2024-09-17T23:29:22.260 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 12/129 2024-09-17T23:29:22.286 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 12/129 2024-09-17T23:29:22.426 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 13/129 2024-09-17T23:29:22.457 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 13/129 2024-09-17T23:29:22.565 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : re2-1:20211101-20.el9.x86_64 14/129 2024-09-17T23:29:22.570 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 92/129 2024-09-17T23:29:22.612 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 93/129 2024-09-17T23:29:22.665 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 15/129 2024-09-17T23:29:22.705 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 93/129 2024-09-17T23:29:22.705 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T23:29:22.705 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T23:29:22.706 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:29:22.706 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 16/129 2024-09-17T23:29:22.741 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.7-2.el9.x86_64 17/129 2024-09-17T23:29:22.756 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 94/129 2024-09-17T23:29:22.816 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 18/129 2024-09-17T23:29:22.867 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 19/129 2024-09-17T23:29:22.908 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-packaging-20.9-5.el9.noarch 20/129 2024-09-17T23:29:22.951 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 21/129 2024-09-17T23:29:22.983 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 22/129 2024-09-17T23:29:23.019 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-5.4.4-4.el9.x86_64 23/129 2024-09-17T23:29:23.091 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 24/129 2024-09-17T23:29:23.143 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 25/129 2024-09-17T23:29:23.186 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 26/129 2024-09-17T23:29:23.233 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 27/129 2024-09-17T23:29:23.311 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 28/129 2024-09-17T23:29:23.344 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x 29/129 2024-09-17T23:29:23.387 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9 30/129 2024-09-17T23:29:23.415 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 31/129 2024-09-17T23:29:23.488 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 32/129 2024-09-17T23:29:23.531 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : luarocks-3.9.2-5.el9.noarch 33/129 2024-09-17T23:29:23.637 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 34/129 2024-09-17T23:29:23.785 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 35/129 2024-09-17T23:29:23.842 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 36/129 2024-09-17T23:29:24.385 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rsa-4.9-2.el9.noarch 37/129 2024-09-17T23:29:24.471 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 38/129 2024-09-17T23:29:24.499 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 39/129 2024-09-17T23:29:24.542 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 40/129 2024-09-17T23:29:24.579 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 41/129 2024-09-17T23:29:24.612 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 42/129 2024-09-17T23:29:24.667 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 43/129 2024-09-17T23:29:24.741 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 44/129 2024-09-17T23:29:24.780 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 45/129 2024-09-17T23:29:24.825 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 46/129 2024-09-17T23:29:24.871 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 47/129 2024-09-17T23:29:24.917 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 48/129 2024-09-17T23:29:24.959 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 49/129 2024-09-17T23:29:25.100 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 50/129 2024-09-17T23:29:25.136 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 51/129 2024-09-17T23:29:25.178 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 52/129 2024-09-17T23:29:25.281 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 53/129 2024-09-17T23:29:25.328 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 54/129 2024-09-17T23:29:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 55/129 2024-09-17T23:29:25.401 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 56/129 2024-09-17T23:29:25.426 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 57/129 2024-09-17T23:29:25.532 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 58/129 2024-09-17T23:29:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 59/129 2024-09-17T23:29:25.708 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 60/129 2024-09-17T23:29:25.765 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 61/129 2024-09-17T23:29:25.807 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 62/129 2024-09-17T23:29:25.845 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 63/129 2024-09-17T23:29:25.888 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 64/129 2024-09-17T23:29:25.921 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 65/129 2024-09-17T23:29:25.963 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 66/129 2024-09-17T23:29:25.997 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 67/129 2024-09-17T23:29:26.085 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 68/129 2024-09-17T23:29:26.137 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 69/129 2024-09-17T23:29:26.168 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 70/129 2024-09-17T23:29:26.273 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 71/129 2024-09-17T23:29:26.336 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 72/129 2024-09-17T23:29:27.071 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 73/129 2024-09-17T23:29:27.124 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 74/129 2024-09-17T23:29:27.225 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 75/129 2024-09-17T23:29:27.395 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 76/129 2024-09-17T23:29:27.445 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 77/129 2024-09-17T23:29:27.629 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-0.3.26-2.el9.x86_64 78/129 2024-09-17T23:29:27.671 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 79/129 2024-09-17T23:29:27.756 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 80/129 2024-09-17T23:29:28.427 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 81/129 2024-09-17T23:29:28.542 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 82/129 2024-09-17T23:29:29.849 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/129 2024-09-17T23:29:29.926 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 84/129 2024-09-17T23:29:30.251 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 85/129 2024-09-17T23:29:30.296 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 86/129 2024-09-17T23:29:30.382 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 86/129 2024-09-17T23:29:30.410 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 87/129 2024-09-17T23:29:30.445 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 88/129 2024-09-17T23:29:30.928 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 89/129 2024-09-17T23:29:30.957 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 90/129 2024-09-17T23:29:31.029 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 90/129 2024-09-17T23:29:31.059 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 91/129 2024-09-17T23:29:33.388 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 92/129 2024-09-17T23:29:33.422 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 92/129 2024-09-17T23:29:33.986 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 92/129 2024-09-17T23:29:34.025 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 93/129 2024-09-17T23:29:34.115 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 93/129 2024-09-17T23:29:34.116 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-17T23:29:34.116 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-17T23:29:34.116 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:29:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 94/129 2024-09-17T23:29:34.336 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 94/129 2024-09-17T23:29:34.336 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /sys 2024-09-17T23:29:34.336 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /proc 2024-09-17T23:29:34.337 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /mnt 2024-09-17T23:29:34.337 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /var/tmp 2024-09-17T23:29:34.337 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /home 2024-09-17T23:29:34.337 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /root 2024-09-17T23:29:34.337 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /tmp 2024-09-17T23:29:34.337 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:29:34.520 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 95/129 2024-09-17T23:29:34.533 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 95/129 2024-09-17T23:29:34.533 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:34.533 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T23:29:34.533 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:29:36.037 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 96/129 2024-09-17T23:29:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 96/129 2024-09-17T23:29:36.064 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:36.064 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T23:29:36.064 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T23:29:36.065 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T23:29:36.065 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:29:36.287 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 97/129 2024-09-17T23:29:36.311 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 97/129 2024-09-17T23:29:36.311 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:36.311 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T23:29:36.311 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T23:29:36.311 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T23:29:36.311 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:29:36.755 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 98/129 2024-09-17T23:29:36.778 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 98/129 2024-09-17T23:29:36.778 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:36.778 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T23:29:36.779 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T23:29:36.779 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T23:29:36.779 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:29:36.848 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ply-3.11-14.el9.noarch 99/129 2024-09-17T23:29:36.903 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 100/129 2024-09-17T23:29:37.089 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 101/129 2024-09-17T23:29:37.155 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 102/129 2024-09-17T23:29:37.232 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 103/129 2024-09-17T23:29:37.329 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 104/129 2024-09-17T23:29:37.468 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 105/129 2024-09-17T23:29:37.514 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 106/129 2024-09-17T23:29:37.545 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 107/129 2024-09-17T23:29:37.573 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : fuse-2.9.9-16.el9.x86_64 108/129 2024-09-17T23:29:38.002 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 109/129 2024-09-17T23:29:38.030 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 109/129 2024-09-17T23:29:38.062 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el 110/129 2024-09-17T23:29:38.124 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.e 111/129 2024-09-17T23:29:38.153 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noar 112/129 2024-09-17T23:29:39.202 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noar 112/129 2024-09-17T23:29:39.239 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.no 113/129 2024-09-17T23:29:40.206 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.no 113/129 2024-09-17T23:29:40.239 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-330.gc2dd 114/129 2024-09-17T23:29:40.342 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-330.gc2dd 114/129 2024-09-17T23:29:40.510 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9 115/129 2024-09-17T23:29:40.541 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 116/129 2024-09-17T23:29:40.566 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 116/129 2024-09-17T23:29:40.566 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:40.566 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T23:29:40.566 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T23:29:40.566 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T23:29:40.566 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:29:40.617 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 117/129 2024-09-17T23:29:40.630 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 117/129 2024-09-17T23:29:40.737 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 118/129 2024-09-17T23:29:41.669 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 119/129 2024-09-17T23:29:41.702 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 120/129 2024-09-17T23:29:41.726 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 120/129 2024-09-17T23:29:41.726 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:41.726 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T23:29:41.726 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T23:29:41.726 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T23:29:41.726 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:29:41.769 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-330.gc2ddf9 121/129 2024-09-17T23:29:41.793 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-330.gc2ddf9 121/129 2024-09-17T23:29:41.793 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:41.793 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T23:29:41.793 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:29:42.104 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 122/129 2024-09-17T23:29:42.127 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 122/129 2024-09-17T23:29:42.127 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:42.127 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T23:29:42.128 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T23:29:42.128 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T23:29:42.128 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:29:46.032 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 94/129 2024-09-17T23:29:46.032 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2024-09-17T23:29:46.032 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2024-09-17T23:29:46.033 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2024-09-17T23:29:46.033 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2024-09-17T23:29:46.033 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2024-09-17T23:29:46.033 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /root 2024-09-17T23:29:46.033 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2024-09-17T23:29:46.033 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:29:46.213 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 95/129 2024-09-17T23:29:46.233 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 95/129 2024-09-17T23:29:46.233 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:46.233 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-17T23:29:46.233 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:29:46.708 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 123/129 2024-09-17T23:29:46.749 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 124/129 2024-09-17T23:29:46.780 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 125/129 2024-09-17T23:29:46.825 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_6 126/129 2024-09-17T23:29:46.853 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : pv-1.6.20-1.el9.x86_64 127/129 2024-09-17T23:29:46.854 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 128/129 2024-09-17T23:29:46.946 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 128/129 2024-09-17T23:29:46.946 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 129/129 2024-09-17T23:29:47.740 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 96/129 2024-09-17T23:29:47.767 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 96/129 2024-09-17T23:29:47.767 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:47.767 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-17T23:29:47.767 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T23:29:47.767 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-17T23:29:47.767 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:29:47.998 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 97/129 2024-09-17T23:29:48.022 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 97/129 2024-09-17T23:29:48.022 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:48.022 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-17T23:29:48.023 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T23:29:48.023 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-17T23:29:48.023 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:29:48.466 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 98/129 2024-09-17T23:29:48.490 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 98/129 2024-09-17T23:29:48.490 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:48.490 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-17T23:29:48.490 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T23:29:48.490 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-17T23:29:48.490 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:29:48.560 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ply-3.11-14.el9.noarch 99/129 2024-09-17T23:29:48.615 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 100/129 2024-09-17T23:29:48.808 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 101/129 2024-09-17T23:29:48.874 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 102/129 2024-09-17T23:29:48.943 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 103/129 2024-09-17T23:29:49.032 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 104/129 2024-09-17T23:29:49.173 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 105/129 2024-09-17T23:29:49.216 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 106/129 2024-09-17T23:29:49.247 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 107/129 2024-09-17T23:29:49.275 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.9-16.el9.x86_64 108/129 2024-09-17T23:29:49.719 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 109/129 2024-09-17T23:29:49.748 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 109/129 2024-09-17T23:29:49.779 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el 110/129 2024-09-17T23:29:49.842 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.e 111/129 2024-09-17T23:29:49.871 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noar 112/129 2024-09-17T23:29:50.910 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noar 112/129 2024-09-17T23:29:50.948 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.no 113/129 2024-09-17T23:29:51.912 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.no 113/129 2024-09-17T23:29:51.932 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-330.gc2dd 114/129 2024-09-17T23:29:52.033 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-330.gc2dd 114/129 2024-09-17T23:29:52.194 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9 115/129 2024-09-17T23:29:52.234 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 116/129 2024-09-17T23:29:52.257 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 116/129 2024-09-17T23:29:52.257 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:52.258 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-17T23:29:52.258 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T23:29:52.258 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-17T23:29:52.258 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:29:52.309 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 117/129 2024-09-17T23:29:52.321 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 117/129 2024-09-17T23:29:52.347 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 129/129 2024-09-17T23:29:52.348 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 1/129 2024-09-17T23:29:52.348 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 2/129 2024-09-17T23:29:52.348 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 3/129 2024-09-17T23:29:52.348 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 4/129 2024-09-17T23:29:52.348 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-330.gc2ddf9 5/129 2024-09-17T23:29:52.348 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 6/129 2024-09-17T23:29:52.348 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 7/129 2024-09-17T23:29:52.348 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 8/129 2024-09-17T23:29:52.348 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 9/129 2024-09-17T23:29:52.348 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 10/129 2024-09-17T23:29:52.348 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 11/129 2024-09-17T23:29:52.349 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 12/129 2024-09-17T23:29:52.349 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_6 13/129 2024-09-17T23:29:52.349 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 14/129 2024-09-17T23:29:52.349 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 15/129 2024-09-17T23:29:52.349 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 16/129 2024-09-17T23:29:52.349 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_ 17/129 2024-09-17T23:29:52.349 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 18/129 2024-09-17T23:29:52.349 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9 19/129 2024-09-17T23:29:52.349 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x 20/129 2024-09-17T23:29:52.349 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 21/129 2024-09-17T23:29:52.349 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 22/129 2024-09-17T23:29:52.349 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 23/129 2024-09-17T23:29:52.350 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 24/129 2024-09-17T23:29:52.350 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 25/129 2024-09-17T23:29:52.350 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 26/129 2024-09-17T23:29:52.350 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 27/129 2024-09-17T23:29:52.359 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.e 28/129 2024-09-17T23:29:52.359 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noar 29/129 2024-09-17T23:29:52.359 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.no 30/129 2024-09-17T23:29:52.359 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-330.gc2dd 31/129 2024-09-17T23:29:52.359 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9 32/129 2024-09-17T23:29:52.359 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 33/129 2024-09-17T23:29:52.359 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el 34/129 2024-09-17T23:29:52.360 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 35/129 2024-09-17T23:29:52.360 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 36/129 2024-09-17T23:29:52.360 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 37/129 2024-09-17T23:29:52.360 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 38/129 2024-09-17T23:29:52.360 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 39/129 2024-09-17T23:29:52.360 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 40/129 2024-09-17T23:29:52.360 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 41/129 2024-09-17T23:29:52.360 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ply-3.11-14.el9.noarch 42/129 2024-09-17T23:29:52.360 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 43/129 2024-09-17T23:29:52.360 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 44/129 2024-09-17T23:29:52.360 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 45/129 2024-09-17T23:29:52.360 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 46/129 2024-09-17T23:29:52.361 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 47/129 2024-09-17T23:29:52.361 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 48/129 2024-09-17T23:29:52.361 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 49/129 2024-09-17T23:29:52.361 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 50/129 2024-09-17T23:29:52.361 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 51/129 2024-09-17T23:29:52.361 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 52/129 2024-09-17T23:29:52.361 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/129 2024-09-17T23:29:52.361 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/129 2024-09-17T23:29:52.361 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 55/129 2024-09-17T23:29:52.361 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 56/129 2024-09-17T23:29:52.361 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 57/129 2024-09-17T23:29:52.361 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 58/129 2024-09-17T23:29:52.362 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 59/129 2024-09-17T23:29:52.362 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 60/129 2024-09-17T23:29:52.362 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/129 2024-09-17T23:29:52.362 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/129 2024-09-17T23:29:52.362 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/129 2024-09-17T23:29:52.362 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/129 2024-09-17T23:29:52.362 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/129 2024-09-17T23:29:52.362 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/129 2024-09-17T23:29:52.362 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/129 2024-09-17T23:29:52.362 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/129 2024-09-17T23:29:52.362 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/129 2024-09-17T23:29:52.363 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/129 2024-09-17T23:29:52.363 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/129 2024-09-17T23:29:52.363 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/129 2024-09-17T23:29:52.363 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/129 2024-09-17T23:29:52.363 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/129 2024-09-17T23:29:52.363 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/129 2024-09-17T23:29:52.363 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/129 2024-09-17T23:29:52.363 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 77/129 2024-09-17T23:29:52.363 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 78/129 2024-09-17T23:29:52.363 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 79/129 2024-09-17T23:29:52.363 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 80/129 2024-09-17T23:29:52.364 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 81/129 2024-09-17T23:29:52.364 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 82/129 2024-09-17T23:29:52.364 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 83/129 2024-09-17T23:29:52.364 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 84/129 2024-09-17T23:29:52.364 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 85/129 2024-09-17T23:29:52.364 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 86/129 2024-09-17T23:29:52.364 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 87/129 2024-09-17T23:29:52.364 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 88/129 2024-09-17T23:29:52.364 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 89/129 2024-09-17T23:29:52.364 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 90/129 2024-09-17T23:29:52.365 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 91/129 2024-09-17T23:29:52.365 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 92/129 2024-09-17T23:29:52.365 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : pv-1.6.20-1.el9.x86_64 93/129 2024-09-17T23:29:52.365 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 94/129 2024-09-17T23:29:52.365 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 95/129 2024-09-17T23:29:52.365 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 96/129 2024-09-17T23:29:52.365 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 97/129 2024-09-17T23:29:52.365 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 98/129 2024-09-17T23:29:52.365 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 99/129 2024-09-17T23:29:52.365 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 100/129 2024-09-17T23:29:52.365 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 101/129 2024-09-17T23:29:52.365 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 102/129 2024-09-17T23:29:52.366 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 103/129 2024-09-17T23:29:52.366 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 104/129 2024-09-17T23:29:52.366 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 105/129 2024-09-17T23:29:52.366 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 106/129 2024-09-17T23:29:52.366 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 107/129 2024-09-17T23:29:52.366 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 108/129 2024-09-17T23:29:52.366 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 109/129 2024-09-17T23:29:52.366 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 110/129 2024-09-17T23:29:52.366 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 111/129 2024-09-17T23:29:52.366 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 112/129 2024-09-17T23:29:52.366 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 113/129 2024-09-17T23:29:52.366 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 114/129 2024-09-17T23:29:52.366 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 115/129 2024-09-17T23:29:52.367 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 116/129 2024-09-17T23:29:52.367 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 117/129 2024-09-17T23:29:52.367 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 118/129 2024-09-17T23:29:52.367 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 119/129 2024-09-17T23:29:52.367 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 120/129 2024-09-17T23:29:52.367 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 121/129 2024-09-17T23:29:52.367 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 122/129 2024-09-17T23:29:52.367 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 123/129 2024-09-17T23:29:52.367 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : re2-1:20211101-20.el9.x86_64 124/129 2024-09-17T23:29:52.367 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 125/129 2024-09-17T23:29:52.367 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 126/129 2024-09-17T23:29:52.367 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 127/129 2024-09-17T23:29:52.367 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 128/129 2024-09-17T23:29:52.421 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 118/129 2024-09-17T23:29:53.352 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 119/129 2024-09-17T23:29:53.394 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 120/129 2024-09-17T23:29:53.419 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 120/129 2024-09-17T23:29:53.419 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:53.419 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-17T23:29:53.419 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T23:29:53.419 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-17T23:29:53.419 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:29:53.462 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-330.gc2ddf9 121/129 2024-09-17T23:29:53.486 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-330.gc2ddf9 121/129 2024-09-17T23:29:53.486 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:53.486 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-17T23:29:53.486 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:29:53.804 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 122/129 2024-09-17T23:29:53.828 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 122/129 2024-09-17T23:29:53.828 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-17T23:29:53.828 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-17T23:29:53.828 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T23:29:53.829 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-17T23:29:53.829 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:29:54.367 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 129/129 2024-09-17T23:29:54.368 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:29:54.368 INFO:teuthology.orchestra.run.smithi096.stdout:Upgraded: 2024-09-17T23:29:54.368 INFO:teuthology.orchestra.run.smithi096.stdout: librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.368 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.368 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-17T23:29:54.368 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T23:29:54.368 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T23:29:54.368 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.368 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.368 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.368 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:29:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:29:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:29:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:29:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:29:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:29:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.369 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:29:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:29:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:29:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T23:29:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T23:29:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T23:29:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-17T23:29:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T23:29:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T23:29:54.370 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T23:29:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T23:29:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T23:29:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T23:29:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T23:29:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T23:29:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T23:29:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.371 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T23:29:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T23:29:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T23:29:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T23:29:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T23:29:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T23:29:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T23:29:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T23:29:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T23:29:54.372 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T23:29:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T23:29:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: pv-1.6.20-1.el9.x86_64 2024-09-17T23:29:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T23:29:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T23:29:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T23:29:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T23:29:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T23:29:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T23:29:54.373 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T23:29:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T23:29:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T23:29:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T23:29:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T23:29:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T23:29:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T23:29:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T23:29:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T23:29:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T23:29:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T23:29:54.374 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T23:29:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T23:29:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T23:29:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T23:29:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T23:29:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T23:29:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T23:29:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T23:29:54.375 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T23:29:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T23:29:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T23:29:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T23:29:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T23:29:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T23:29:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T23:29:54.376 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T23:29:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T23:29:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T23:29:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T23:29:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T23:29:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T23:29:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T23:29:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T23:29:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T23:29:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T23:29:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T23:29:54.377 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T23:29:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T23:29:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T23:29:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T23:29:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T23:29:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T23:29:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T23:29:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T23:29:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T23:29:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T23:29:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T23:29:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T23:29:54.378 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T23:29:54.379 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T23:29:54.379 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T23:29:54.379 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.379 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.379 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:29:54.379 INFO:teuthology.orchestra.run.smithi096.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T23:29:54.379 INFO:teuthology.orchestra.run.smithi096.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T23:29:54.379 INFO:teuthology.orchestra.run.smithi096.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T23:29:54.379 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T23:29:54.379 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:29:54.380 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-17T23:29:54.698 DEBUG:teuthology.parallel:result is None 2024-09-17T23:29:58.425 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 123/129 2024-09-17T23:29:58.465 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 124/129 2024-09-17T23:29:58.497 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 125/129 2024-09-17T23:29:58.538 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_6 126/129 2024-09-17T23:29:58.570 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : pv-1.6.20-1.el9.x86_64 127/129 2024-09-17T23:29:58.570 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 128/129 2024-09-17T23:29:58.654 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 128/129 2024-09-17T23:29:58.654 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 129/129 2024-09-17T23:30:04.089 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 129/129 2024-09-17T23:30:04.089 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 1/129 2024-09-17T23:30:04.089 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 2/129 2024-09-17T23:30:04.089 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 3/129 2024-09-17T23:30:04.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 4/129 2024-09-17T23:30:04.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-330.gc2ddf9 5/129 2024-09-17T23:30:04.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 6/129 2024-09-17T23:30:04.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 7/129 2024-09-17T23:30:04.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 8/129 2024-09-17T23:30:04.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 9/129 2024-09-17T23:30:04.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 10/129 2024-09-17T23:30:04.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 11/129 2024-09-17T23:30:04.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 12/129 2024-09-17T23:30:04.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_6 13/129 2024-09-17T23:30:04.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 14/129 2024-09-17T23:30:04.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 15/129 2024-09-17T23:30:04.090 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 16/129 2024-09-17T23:30:04.091 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_ 17/129 2024-09-17T23:30:04.091 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 18/129 2024-09-17T23:30:04.091 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9 19/129 2024-09-17T23:30:04.091 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x 20/129 2024-09-17T23:30:04.091 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 21/129 2024-09-17T23:30:04.091 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 22/129 2024-09-17T23:30:04.091 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 23/129 2024-09-17T23:30:04.091 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 24/129 2024-09-17T23:30:04.091 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 25/129 2024-09-17T23:30:04.091 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 26/129 2024-09-17T23:30:04.091 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 27/129 2024-09-17T23:30:04.091 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.e 28/129 2024-09-17T23:30:04.092 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noar 29/129 2024-09-17T23:30:04.092 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.no 30/129 2024-09-17T23:30:04.092 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-330.gc2dd 31/129 2024-09-17T23:30:04.092 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9 32/129 2024-09-17T23:30:04.092 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 33/129 2024-09-17T23:30:04.092 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el 34/129 2024-09-17T23:30:04.092 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 35/129 2024-09-17T23:30:04.092 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 36/129 2024-09-17T23:30:04.092 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 37/129 2024-09-17T23:30:04.092 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 38/129 2024-09-17T23:30:04.092 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 39/129 2024-09-17T23:30:04.092 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 40/129 2024-09-17T23:30:04.093 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 41/129 2024-09-17T23:30:04.093 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ply-3.11-14.el9.noarch 42/129 2024-09-17T23:30:04.093 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 43/129 2024-09-17T23:30:04.093 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 44/129 2024-09-17T23:30:04.093 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 45/129 2024-09-17T23:30:04.093 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 46/129 2024-09-17T23:30:04.093 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 47/129 2024-09-17T23:30:04.093 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 48/129 2024-09-17T23:30:04.093 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 49/129 2024-09-17T23:30:04.093 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 50/129 2024-09-17T23:30:04.093 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 51/129 2024-09-17T23:30:04.093 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 52/129 2024-09-17T23:30:04.093 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/129 2024-09-17T23:30:04.094 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/129 2024-09-17T23:30:04.094 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 55/129 2024-09-17T23:30:04.094 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 56/129 2024-09-17T23:30:04.094 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 57/129 2024-09-17T23:30:04.094 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 58/129 2024-09-17T23:30:04.094 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 59/129 2024-09-17T23:30:04.094 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 60/129 2024-09-17T23:30:04.094 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/129 2024-09-17T23:30:04.094 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/129 2024-09-17T23:30:04.094 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/129 2024-09-17T23:30:04.094 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/129 2024-09-17T23:30:04.094 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/129 2024-09-17T23:30:04.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/129 2024-09-17T23:30:04.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/129 2024-09-17T23:30:04.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/129 2024-09-17T23:30:04.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/129 2024-09-17T23:30:04.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/129 2024-09-17T23:30:04.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/129 2024-09-17T23:30:04.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/129 2024-09-17T23:30:04.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/129 2024-09-17T23:30:04.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/129 2024-09-17T23:30:04.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/129 2024-09-17T23:30:04.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/129 2024-09-17T23:30:04.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 77/129 2024-09-17T23:30:04.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 78/129 2024-09-17T23:30:04.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 79/129 2024-09-17T23:30:04.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 80/129 2024-09-17T23:30:04.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 81/129 2024-09-17T23:30:04.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 82/129 2024-09-17T23:30:04.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 83/129 2024-09-17T23:30:04.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 84/129 2024-09-17T23:30:04.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 85/129 2024-09-17T23:30:04.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 86/129 2024-09-17T23:30:04.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 87/129 2024-09-17T23:30:04.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 88/129 2024-09-17T23:30:04.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 89/129 2024-09-17T23:30:04.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 90/129 2024-09-17T23:30:04.096 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 91/129 2024-09-17T23:30:04.097 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 92/129 2024-09-17T23:30:04.097 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : pv-1.6.20-1.el9.x86_64 93/129 2024-09-17T23:30:04.097 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 94/129 2024-09-17T23:30:04.097 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 95/129 2024-09-17T23:30:04.097 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 96/129 2024-09-17T23:30:04.097 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 97/129 2024-09-17T23:30:04.097 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 98/129 2024-09-17T23:30:04.097 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 99/129 2024-09-17T23:30:04.097 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 100/129 2024-09-17T23:30:04.097 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 101/129 2024-09-17T23:30:04.098 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 102/129 2024-09-17T23:30:04.098 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 103/129 2024-09-17T23:30:04.098 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 104/129 2024-09-17T23:30:04.098 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 105/129 2024-09-17T23:30:04.098 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 106/129 2024-09-17T23:30:04.098 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 107/129 2024-09-17T23:30:04.098 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 108/129 2024-09-17T23:30:04.098 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 109/129 2024-09-17T23:30:04.098 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 110/129 2024-09-17T23:30:04.098 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 111/129 2024-09-17T23:30:04.098 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 112/129 2024-09-17T23:30:04.098 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 113/129 2024-09-17T23:30:04.098 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 114/129 2024-09-17T23:30:04.098 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 115/129 2024-09-17T23:30:04.099 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 116/129 2024-09-17T23:30:04.099 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 117/129 2024-09-17T23:30:04.099 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 118/129 2024-09-17T23:30:04.099 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 119/129 2024-09-17T23:30:04.099 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 120/129 2024-09-17T23:30:04.099 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 121/129 2024-09-17T23:30:04.099 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 122/129 2024-09-17T23:30:04.099 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 123/129 2024-09-17T23:30:04.099 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : re2-1:20211101-20.el9.x86_64 124/129 2024-09-17T23:30:04.099 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 125/129 2024-09-17T23:30:04.099 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 126/129 2024-09-17T23:30:04.099 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 127/129 2024-09-17T23:30:04.099 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 128/129 2024-09-17T23:30:06.350 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 129/129 2024-09-17T23:30:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:06.351 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2024-09-17T23:30:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.351 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-09-17T23:30:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-17T23:30:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-17T23:30:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.351 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.352 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.352 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:30:06.352 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.352 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.352 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.352 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:30:06.352 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:30:06.352 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:30:06.352 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:30:06.352 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:30:06.352 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:30:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:30:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:19.1.1-330.gc2ddf972.el9.noarch 2024-09-17T23:30:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-17T23:30:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-17T23:30:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-17T23:30:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-17T23:30:06.353 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-17T23:30:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-17T23:30:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-17T23:30:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-17T23:30:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-17T23:30:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-17T23:30:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-17T23:30:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-17T23:30:06.354 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-17T23:30:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-17T23:30:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-17T23:30:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-17T23:30:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: lua-5.4.4-4.el9.x86_64 2024-09-17T23:30:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-17T23:30:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-17T23:30:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-17T23:30:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-17T23:30:06.355 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-17T23:30:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-17T23:30:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-17T23:30:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: pv-1.6.20-1.el9.x86_64 2024-09-17T23:30:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-17T23:30:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-17T23:30:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-17T23:30:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-17T23:30:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-17T23:30:06.356 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-17T23:30:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-17T23:30:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-17T23:30:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-17T23:30:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-17T23:30:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-17T23:30:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-17T23:30:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-17T23:30:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-17T23:30:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-17T23:30:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-17T23:30:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-17T23:30:06.357 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-17T23:30:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-17T23:30:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-17T23:30:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-17T23:30:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-17T23:30:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-17T23:30:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-17T23:30:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-17T23:30:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-17T23:30:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-17T23:30:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-17T23:30:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-17T23:30:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-17T23:30:06.358 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-17T23:30:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-17T23:30:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply-3.11-14.el9.noarch 2024-09-17T23:30:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-17T23:30:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-17T23:30:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-17T23:30:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-17T23:30:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-17T23:30:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-17T23:30:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-17T23:30:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-17T23:30:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.359 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-17T23:30:06.360 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-17T23:30:06.360 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-17T23:30:06.360 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.360 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-17T23:30:06.360 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-17T23:30:06.360 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-17T23:30:06.360 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-17T23:30:06.360 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-17T23:30:06.360 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-17T23:30:06.360 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-17T23:30:06.360 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-17T23:30:06.360 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-17T23:30:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-17T23:30:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-17T23:30:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-17T23:30:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-17T23:30:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-17T23:30:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:19.1.1-330.gc2ddf972.el9.x86_64 2024-09-17T23:30:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: re2-1:20211101-20.el9.x86_64 2024-09-17T23:30:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-17T23:30:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-17T23:30:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-17T23:30:06.361 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:06.362 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-09-17T23:30:06.693 DEBUG:teuthology.parallel:result is None 2024-09-17T23:30:06.693 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T23:30:06.837 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T23:30:06.863 INFO:teuthology.orchestra.run.smithi096.stdout:19.1.1-330.gc2ddf972.el9 2024-09-17T23:30:06.863 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-330.gc2ddf972.el9 2024-09-17T23:30:06.864 INFO:teuthology.task.install:The correct ceph version 19.1.1-330.gc2ddf972 is installed. 2024-09-17T23:30:06.866 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T23:30:07.001 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-17T23:30:07.024 INFO:teuthology.orchestra.run.smithi192.stdout:19.1.1-330.gc2ddf972.el9 2024-09-17T23:30:07.025 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-330.gc2ddf972.el9 2024-09-17T23:30:07.025 INFO:teuthology.task.install:The correct ceph version 19.1.1-330.gc2ddf972 is installed. 2024-09-17T23:30:07.027 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-17T23:30:07.027 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:07.027 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T23:30:07.061 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:07.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-17T23:30:07.093 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-17T23:30:07.093 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:07.093 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T23:30:07.129 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T23:30:07.197 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:07.197 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2024-09-17T23:30:07.222 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-17T23:30:07.287 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-17T23:30:07.288 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:07.288 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T23:30:07.314 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T23:30:07.381 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:07.381 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-17T23:30:07.405 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-17T23:30:07.471 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-17T23:30:07.471 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:07.471 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T23:30:07.496 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T23:30:07.562 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:07.562 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2024-09-17T23:30:07.587 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-17T23:30:07.651 INFO:teuthology.run_tasks:Running task ceph... 2024-09-17T23:30:07.739 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-17T23:30:07.739 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 777 /var/log/ceph 2024-09-17T23:30:07.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /var/log/ceph 2024-09-17T23:30:07.765 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-17T23:30:07.765 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T23:30:07.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-17T23:30:07.832 INFO:tasks.ceph:Creating extra log directories... 2024-09-17T23:30:07.832 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T23:30:07.872 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-17T23:30:07.909 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-17T23:30:07.910 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-17T23:30:07.910 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909591', 'branch': 'squid', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_latest} workloads/compare-mirror-images-krbd}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7909591', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}], 'name': 'teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}, 'extra_system_packages': ['pv']}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b0d356b98e08f6bc59ef15652f0d0e4298fc83c7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 1983, 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f', 'sleep_before_teardown': 0, 'subset': '79/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_b0d356b98e08f6bc59ef15652f0d0e4298fc83c7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'b0d356b98e08f6bc59ef15652f0d0e4298fc83c7', 'targets': {'smithi096.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIvRkATk5dJFdg8bZ75kCPOVVti6XPuOAeNrFoSrWb1IZV+uOrgL1tk+67NoeOcH0c7OiLJ0627Yp39IrVU+X+A=', 'smithi192.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB07Wg4TIz69AdzDYYYnJEbgHMOwtweLxwBPUjQITs/NiMWDkBh0Ms3D/BGVdlo9PZk+OtslZcKDf/1fgkLbihA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f', 'extra_system_packages': ['pv']}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/compare_mirror_images.sh']}, 'env': {'RBD_DEVICE_TYPE': 'krbd'}, 'timeout': '3h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-17T23:30:07.910 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-17T23:30:07.937 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-17T23:30:07.953 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T23:30:07.992 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T23:30:08.017 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:08.017 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T23:30:08.073 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T23:30:08.073 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_1 2024-09-17T23:30:08.128 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T23:30:08.128 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:30:08.128 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-17T23:30:08.128 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:30:08.128 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:30:08.128 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-17 23:29:47.420777415 +0000 2024-09-17T23:30:08.128 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-17 23:28:08.587346650 +0000 2024-09-17T23:30:08.128 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-17 23:28:08.587346650 +0000 2024-09-17T23:30:08.129 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-17 23:28:08.587346650 +0000 2024-09-17T23:30:08.129 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T23:30:08.191 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-17T23:30:08.191 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-17T23:30:08.192 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000256244 s, 2.0 MB/s 2024-09-17T23:30:08.193 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T23:30:08.249 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_2 2024-09-17T23:30:08.304 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T23:30:08.304 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:30:08.304 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-17T23:30:08.304 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:30:08.304 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:30:08.304 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-17 23:29:47.420777415 +0000 2024-09-17T23:30:08.304 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-17 23:28:08.904340019 +0000 2024-09-17T23:30:08.304 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-17 23:28:08.904340019 +0000 2024-09-17T23:30:08.305 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-17 23:28:08.904340019 +0000 2024-09-17T23:30:08.305 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T23:30:08.367 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-17T23:30:08.367 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-17T23:30:08.367 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000171621 s, 3.0 MB/s 2024-09-17T23:30:08.368 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T23:30:08.424 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_3 2024-09-17T23:30:08.479 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T23:30:08.479 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:30:08.479 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-17T23:30:08.479 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:30:08.479 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:30:08.479 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-17 23:29:47.421777394 +0000 2024-09-17T23:30:08.479 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-17 23:28:09.219333430 +0000 2024-09-17T23:30:08.480 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-17 23:28:09.219333430 +0000 2024-09-17T23:30:08.480 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-17 23:28:09.219333430 +0000 2024-09-17T23:30:08.480 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T23:30:08.542 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-17T23:30:08.542 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-17T23:30:08.542 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000238791 s, 2.1 MB/s 2024-09-17T23:30:08.543 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T23:30:08.600 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_4 2024-09-17T23:30:08.655 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T23:30:08.656 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:30:08.656 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-17T23:30:08.656 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:30:08.656 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:30:08.656 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-17 23:29:47.421777394 +0000 2024-09-17T23:30:08.656 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-17 23:28:09.540326716 +0000 2024-09-17T23:30:08.656 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-17 23:28:09.540326716 +0000 2024-09-17T23:30:08.656 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-17 23:28:09.540326716 +0000 2024-09-17T23:30:08.657 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T23:30:08.719 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-17T23:30:08.719 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-17T23:30:08.719 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000234843 s, 2.2 MB/s 2024-09-17T23:30:08.721 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T23:30:08.777 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:30:08.777 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-17T23:30:08.777 INFO:tasks.ceph:Generating config... 2024-09-17T23:30:08.779 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-17T23:30:08.779 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-17T23:30:08.779 INFO:tasks.ceph:[client] log to stderr = False 2024-09-17T23:30:08.779 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-17T23:30:08.779 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:08.779 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:08.779 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:08.779 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:08.779 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:08.779 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:08.779 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:08.779 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:08.779 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:08.780 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:08.780 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:08.780 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:08.780 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:08.780 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:08.780 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-17T23:30:08.780 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-17T23:30:08.780 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-17T23:30:08.780 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-17T23:30:08.780 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-17T23:30:08.780 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-17T23:30:08.780 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-17T23:30:08.780 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-17T23:30:08.781 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-17T23:30:08.781 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-09-17T23:30:08.781 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-17T23:30:08.781 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-17T23:30:08.781 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-17T23:30:08.781 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-17T23:30:08.781 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-17T23:30:08.781 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-17T23:30:08.781 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-17T23:30:08.781 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-17T23:30:08.781 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-17T23:30:08.781 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-17T23:30:08.781 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-17T23:30:08.781 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-17T23:30:08.782 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-17T23:30:08.782 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-17T23:30:08.865 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/cluster1.keyring 2024-09-17T23:30:08.867 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-17T23:30:08.909 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-17T23:30:08.973 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.96')] 2024-09-17T23:30:08.974 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.96', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-17T23:30:08.975 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:08.975 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T23:30:09.029 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.96 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:30:09.109 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:30:09.109 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: generated fsid eeb17cc0-06d7-4ce4-845b-f0ddd9bde342 2024-09-17T23:30:09.109 INFO:teuthology.orchestra.run.smithi096.stdout:setting min_mon_release = quincy 2024-09-17T23:30:09.109 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 0 2024-09-17T23:30:09.110 INFO:teuthology.orchestra.run.smithi096.stdout:fsid eeb17cc0-06d7-4ce4-845b-f0ddd9bde342 2024-09-17T23:30:09.110 INFO:teuthology.orchestra.run.smithi096.stdout:last_changed 2024-09-17T23:30:09.108298+0000 2024-09-17T23:30:09.110 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-09-17T23:30:09.108298+0000 2024-09-17T23:30:09.110 INFO:teuthology.orchestra.run.smithi096.stdout:min_mon_release 17 (quincy) 2024-09-17T23:30:09.110 INFO:teuthology.orchestra.run.smithi096.stdout:election_strategy: 1 2024-09-17T23:30:09.110 INFO:teuthology.orchestra.run.smithi096.stdout:0: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.a 2024-09-17T23:30:09.110 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-17T23:30:09.111 DEBUG:teuthology.orchestra.run.smithi096:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T23:30:09.167 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID eeb17cc0-06d7-4ce4-845b-f0ddd9bde342... 2024-09-17T23:30:09.168 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-17T23:30:09.209 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-17T23:30:09.249 INFO:teuthology.orchestra.run.smithi096.stdout:[global] 2024-09-17T23:30:09.250 INFO:teuthology.orchestra.run.smithi096.stdout: chdir = "" 2024-09-17T23:30:09.250 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T23:30:09.250 INFO:teuthology.orchestra.run.smithi096.stdout: auth supported = cephx 2024-09-17T23:30:09.250 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.250 INFO:teuthology.orchestra.run.smithi096.stdout: filestore xattr use omap = true 2024-09-17T23:30:09.250 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.250 INFO:teuthology.orchestra.run.smithi096.stdout: mon clock drift allowed = 1.000 2024-09-17T23:30:09.250 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.250 INFO:teuthology.orchestra.run.smithi096.stdout: osd crush chooseleaf type = 0 2024-09-17T23:30:09.250 INFO:teuthology.orchestra.run.smithi096.stdout: auth debug = true 2024-09-17T23:30:09.250 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.250 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on old message = true 2024-09-17T23:30:09.251 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on bug = true 2024-09-17T23:30:09.251 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.251 INFO:teuthology.orchestra.run.smithi096.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T23:30:09.251 INFO:teuthology.orchestra.run.smithi096.stdout: mon pg warn max object skew = 0 2024-09-17T23:30:09.251 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.251 INFO:teuthology.orchestra.run.smithi096.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T23:30:09.251 INFO:teuthology.orchestra.run.smithi096.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T23:30:09.251 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.251 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default size = 2 2024-09-17T23:30:09.251 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.251 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow primary affinity = true 2024-09-17T23:30:09.251 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow pg remap = true 2024-09-17T23:30:09.251 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on legacy crush tunables = false 2024-09-17T23:30:09.252 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on crush straw calc version zero = false 2024-09-17T23:30:09.252 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on no sortbitwise = false 2024-09-17T23:30:09.252 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on osd down out interval zero = false 2024-09-17T23:30:09.252 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on too few osds = false 2024-09-17T23:30:09.252 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T23:30:09.252 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T23:30:09.252 INFO:teuthology.orchestra.run.smithi096.stdout: mon_allow_pool_size_one = true 2024-09-17T23:30:09.252 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.252 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T23:30:09.252 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.252 INFO:teuthology.orchestra.run.smithi096.stdout: osd default data pool replay window = 5 2024-09-17T23:30:09.252 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.252 INFO:teuthology.orchestra.run.smithi096.stdout: mon allow pool delete = true 2024-09-17T23:30:09.253 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.253 INFO:teuthology.orchestra.run.smithi096.stdout: mon cluster log file level = debug 2024-09-17T23:30:09.253 INFO:teuthology.orchestra.run.smithi096.stdout: debug asserts on shutdown = true 2024-09-17T23:30:09.253 INFO:teuthology.orchestra.run.smithi096.stdout: mon health detail to clog = false 2024-09-17T23:30:09.253 INFO:teuthology.orchestra.run.smithi096.stdout: mon host = 172.21.15.96 2024-09-17T23:30:09.253 INFO:teuthology.orchestra.run.smithi096.stdout: mon client directed command retry = 5 2024-09-17T23:30:09.253 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on pool no app = False 2024-09-17T23:30:09.253 INFO:teuthology.orchestra.run.smithi096.stdout: ms inject socket failures = 5000 2024-09-17T23:30:09.253 INFO:teuthology.orchestra.run.smithi096.stdout: fsid = eeb17cc0-06d7-4ce4-845b-f0ddd9bde342 2024-09-17T23:30:09.253 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.253 INFO:teuthology.orchestra.run.smithi096.stdout:[osd] 2024-09-17T23:30:09.253 INFO:teuthology.orchestra.run.smithi096.stdout: osd journal size = 100 2024-09-17T23:30:09.253 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.253 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub load threshold = 5.0 2024-09-17T23:30:09.254 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub max interval = 600 2024-09-17T23:30:09.254 INFO:teuthology.orchestra.run.smithi096.stdout: osd mclock profile = high_recovery_ops 2024-09-17T23:30:09.254 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.254 INFO:teuthology.orchestra.run.smithi096.stdout: osd recover clone overlap = true 2024-09-17T23:30:09.254 INFO:teuthology.orchestra.run.smithi096.stdout: osd recovery max chunk = 1048576 2024-09-17T23:30:09.254 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.254 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug shutdown = true 2024-09-17T23:30:09.254 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug op order = true 2024-09-17T23:30:09.254 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify stray on activate = true 2024-09-17T23:30:09.254 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug trim objects = true 2024-09-17T23:30:09.254 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.254 INFO:teuthology.orchestra.run.smithi096.stdout: osd open classes on start = true 2024-09-17T23:30:09.254 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug pg log writeout = true 2024-09-17T23:30:09.254 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.255 INFO:teuthology.orchestra.run.smithi096.stdout: osd deep scrub update digest min age = 30 2024-09-17T23:30:09.255 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.255 INFO:teuthology.orchestra.run.smithi096.stdout: osd map max advance = 10 2024-09-17T23:30:09.255 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.255 INFO:teuthology.orchestra.run.smithi096.stdout: journal zero on create = true 2024-09-17T23:30:09.255 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.255 INFO:teuthology.orchestra.run.smithi096.stdout: filestore ondisk finisher threads = 3 2024-09-17T23:30:09.255 INFO:teuthology.orchestra.run.smithi096.stdout: filestore apply finisher threads = 3 2024-09-17T23:30:09.255 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.255 INFO:teuthology.orchestra.run.smithi096.stdout: bdev debug aio = true 2024-09-17T23:30:09.255 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug misdirected ops = true 2024-09-17T23:30:09.255 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore block size = 96636764160 2024-09-17T23:30:09.255 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore compression algorithm = zstd 2024-09-17T23:30:09.255 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore compression mode = aggressive 2024-09-17T23:30:09.256 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore fsck on mount = True 2024-09-17T23:30:09.256 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluefs = 1/20 2024-09-17T23:30:09.256 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluestore = 1/20 2024-09-17T23:30:09.256 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-17T23:30:09.256 INFO:teuthology.orchestra.run.smithi096.stdout: debug osd = 20 2024-09-17T23:30:09.256 INFO:teuthology.orchestra.run.smithi096.stdout: debug rocksdb = 4/10 2024-09-17T23:30:09.256 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T23:30:09.256 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd full ratio = 0.9 2024-09-17T23:30:09.256 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd nearfull ratio = 0.8 2024-09-17T23:30:09.256 INFO:teuthology.orchestra.run.smithi096.stdout: osd failsafe full ratio = 0.95 2024-09-17T23:30:09.256 INFO:teuthology.orchestra.run.smithi096.stdout: osd objectstore = bluestore 2024-09-17T23:30:09.256 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.256 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr] 2024-09-17T23:30:09.256 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-17T23:30:09.257 INFO:teuthology.orchestra.run.smithi096.stdout: debug mgr = 20 2024-09-17T23:30:09.257 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-09-17T23:30:09.257 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-09-17T23:30:09.257 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:30:09.257 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:30:09.257 INFO:teuthology.orchestra.run.smithi096.stdout: mgr/telemetry/nag = false 2024-09-17T23:30:09.257 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.257 INFO:teuthology.orchestra.run.smithi096.stdout:[mon] 2024-09-17T23:30:09.257 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-17T23:30:09.257 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-09-17T23:30:09.257 INFO:teuthology.orchestra.run.smithi096.stdout: debug paxos = 20 2024-09-17T23:30:09.257 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-09-17T23:30:09.257 INFO:teuthology.orchestra.run.smithi096.stdout: mon data avail warn = 5 2024-09-17T23:30:09.258 INFO:teuthology.orchestra.run.smithi096.stdout: mon mgr mkfs grace = 240 2024-09-17T23:30:09.258 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:30:09.258 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd reporter subtree level = osd 2024-09-17T23:30:09.258 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd prime pg temp = true 2024-09-17T23:30:09.258 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:30:09.258 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.258 INFO:teuthology.orchestra.run.smithi096.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T23:30:09.258 INFO:teuthology.orchestra.run.smithi096.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T23:30:09.258 INFO:teuthology.orchestra.run.smithi096.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T23:30:09.258 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.259 INFO:teuthology.orchestra.run.smithi096.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T23:30:09.259 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T23:30:09.259 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T23:30:09.259 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.259 INFO:teuthology.orchestra.run.smithi096.stdout: # 1m isn't quite enough 2024-09-17T23:30:09.259 INFO:teuthology.orchestra.run.smithi096.stdout: mon_down_mkfs_grace = 2m 2024-09-17T23:30:09.259 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.259 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_filestore_osds = false 2024-09-17T23:30:09.259 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:09.259 INFO:teuthology.orchestra.run.smithi096.stdout:[client] 2024-09-17T23:30:09.259 INFO:teuthology.orchestra.run.smithi096.stdout: rgw cache enabled = true 2024-09-17T23:30:09.259 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable ops log = true 2024-09-17T23:30:09.259 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable usage log = true 2024-09-17T23:30:09.259 INFO:teuthology.orchestra.run.smithi096.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T23:30:09.260 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T23:30:09.260 INFO:teuthology.orchestra.run.smithi096.stdout: debug rbd = 20 2024-09-17T23:30:09.260 INFO:teuthology.orchestra.run.smithi096.stdout: debug rbd_mirror = 15 2024-09-17T23:30:09.260 INFO:teuthology.orchestra.run.smithi096.stdout: log to stderr = False 2024-09-17T23:30:09.260 INFO:teuthology.orchestra.run.smithi096.stdout: rbd default features = 125 2024-09-17T23:30:09.260 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.a] 2024-09-17T23:30:09.260 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.0] 2024-09-17T23:30:09.260 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:09.260 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:09.260 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.1] 2024-09-17T23:30:09.260 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:09.260 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:09.260 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.2] 2024-09-17T23:30:09.261 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:09.261 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:09.261 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.3] 2024-09-17T23:30:09.261 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:09.261 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:09.261 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.4] 2024-09-17T23:30:09.261 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:09.261 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:09.261 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.5] 2024-09-17T23:30:09.261 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:09.261 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:09.261 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.6] 2024-09-17T23:30:09.261 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:09.261 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:09.262 INFO:teuthology.orchestra.run.smithi192.stdout:[global] 2024-09-17T23:30:09.262 INFO:teuthology.orchestra.run.smithi192.stdout: chdir = "" 2024-09-17T23:30:09.262 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T23:30:09.262 INFO:teuthology.orchestra.run.smithi192.stdout: auth supported = cephx 2024-09-17T23:30:09.262 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.262 INFO:teuthology.orchestra.run.smithi192.stdout: filestore xattr use omap = true 2024-09-17T23:30:09.263 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.263 INFO:teuthology.orchestra.run.smithi192.stdout: mon clock drift allowed = 1.000 2024-09-17T23:30:09.263 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.263 INFO:teuthology.orchestra.run.smithi192.stdout: osd crush chooseleaf type = 0 2024-09-17T23:30:09.263 INFO:teuthology.orchestra.run.smithi192.stdout: auth debug = true 2024-09-17T23:30:09.263 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.263 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on old message = true 2024-09-17T23:30:09.263 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on bug = true 2024-09-17T23:30:09.263 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.263 INFO:teuthology.orchestra.run.smithi192.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T23:30:09.263 INFO:teuthology.orchestra.run.smithi192.stdout: mon pg warn max object skew = 0 2024-09-17T23:30:09.263 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.263 INFO:teuthology.orchestra.run.smithi192.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T23:30:09.264 INFO:teuthology.orchestra.run.smithi192.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T23:30:09.264 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.264 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default size = 2 2024-09-17T23:30:09.264 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.264 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow primary affinity = true 2024-09-17T23:30:09.264 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow pg remap = true 2024-09-17T23:30:09.264 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on legacy crush tunables = false 2024-09-17T23:30:09.264 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on crush straw calc version zero = false 2024-09-17T23:30:09.264 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on no sortbitwise = false 2024-09-17T23:30:09.265 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on osd down out interval zero = false 2024-09-17T23:30:09.265 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on too few osds = false 2024-09-17T23:30:09.265 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T23:30:09.265 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T23:30:09.265 INFO:teuthology.orchestra.run.smithi192.stdout: mon_allow_pool_size_one = true 2024-09-17T23:30:09.265 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.265 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T23:30:09.265 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.265 INFO:teuthology.orchestra.run.smithi192.stdout: osd default data pool replay window = 5 2024-09-17T23:30:09.265 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.265 INFO:teuthology.orchestra.run.smithi192.stdout: mon allow pool delete = true 2024-09-17T23:30:09.265 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.265 INFO:teuthology.orchestra.run.smithi192.stdout: mon cluster log file level = debug 2024-09-17T23:30:09.265 INFO:teuthology.orchestra.run.smithi192.stdout: debug asserts on shutdown = true 2024-09-17T23:30:09.266 INFO:teuthology.orchestra.run.smithi192.stdout: mon health detail to clog = false 2024-09-17T23:30:09.266 INFO:teuthology.orchestra.run.smithi192.stdout: mon host = 172.21.15.96 2024-09-17T23:30:09.266 INFO:teuthology.orchestra.run.smithi192.stdout: mon client directed command retry = 5 2024-09-17T23:30:09.266 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on pool no app = False 2024-09-17T23:30:09.266 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject socket failures = 5000 2024-09-17T23:30:09.266 INFO:teuthology.orchestra.run.smithi192.stdout: fsid = eeb17cc0-06d7-4ce4-845b-f0ddd9bde342 2024-09-17T23:30:09.266 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.266 INFO:teuthology.orchestra.run.smithi192.stdout:[osd] 2024-09-17T23:30:09.266 INFO:teuthology.orchestra.run.smithi192.stdout: osd journal size = 100 2024-09-17T23:30:09.266 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.266 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub load threshold = 5.0 2024-09-17T23:30:09.266 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub max interval = 600 2024-09-17T23:30:09.266 INFO:teuthology.orchestra.run.smithi192.stdout: osd mclock profile = high_recovery_ops 2024-09-17T23:30:09.267 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.267 INFO:teuthology.orchestra.run.smithi192.stdout: osd recover clone overlap = true 2024-09-17T23:30:09.267 INFO:teuthology.orchestra.run.smithi192.stdout: osd recovery max chunk = 1048576 2024-09-17T23:30:09.267 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.267 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug shutdown = true 2024-09-17T23:30:09.267 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug op order = true 2024-09-17T23:30:09.267 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify stray on activate = true 2024-09-17T23:30:09.267 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug trim objects = true 2024-09-17T23:30:09.267 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.267 INFO:teuthology.orchestra.run.smithi192.stdout: osd open classes on start = true 2024-09-17T23:30:09.267 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug pg log writeout = true 2024-09-17T23:30:09.267 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.267 INFO:teuthology.orchestra.run.smithi192.stdout: osd deep scrub update digest min age = 30 2024-09-17T23:30:09.267 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.268 INFO:teuthology.orchestra.run.smithi192.stdout: osd map max advance = 10 2024-09-17T23:30:09.268 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.268 INFO:teuthology.orchestra.run.smithi192.stdout: journal zero on create = true 2024-09-17T23:30:09.268 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.268 INFO:teuthology.orchestra.run.smithi192.stdout: filestore ondisk finisher threads = 3 2024-09-17T23:30:09.268 INFO:teuthology.orchestra.run.smithi192.stdout: filestore apply finisher threads = 3 2024-09-17T23:30:09.268 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.268 INFO:teuthology.orchestra.run.smithi192.stdout: bdev debug aio = true 2024-09-17T23:30:09.268 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug misdirected ops = true 2024-09-17T23:30:09.268 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore block size = 96636764160 2024-09-17T23:30:09.268 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore compression algorithm = zstd 2024-09-17T23:30:09.268 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore compression mode = aggressive 2024-09-17T23:30:09.268 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore fsck on mount = True 2024-09-17T23:30:09.269 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluefs = 1/20 2024-09-17T23:30:09.269 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluestore = 1/20 2024-09-17T23:30:09.269 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-09-17T23:30:09.269 INFO:teuthology.orchestra.run.smithi192.stdout: debug osd = 20 2024-09-17T23:30:09.269 INFO:teuthology.orchestra.run.smithi192.stdout: debug rocksdb = 4/10 2024-09-17T23:30:09.269 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T23:30:09.269 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd full ratio = 0.9 2024-09-17T23:30:09.269 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd nearfull ratio = 0.8 2024-09-17T23:30:09.269 INFO:teuthology.orchestra.run.smithi192.stdout: osd failsafe full ratio = 0.95 2024-09-17T23:30:09.269 INFO:teuthology.orchestra.run.smithi192.stdout: osd objectstore = bluestore 2024-09-17T23:30:09.269 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.269 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr] 2024-09-17T23:30:09.269 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-09-17T23:30:09.270 INFO:teuthology.orchestra.run.smithi192.stdout: debug mgr = 20 2024-09-17T23:30:09.270 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-09-17T23:30:09.270 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-09-17T23:30:09.270 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:30:09.270 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:30:09.270 INFO:teuthology.orchestra.run.smithi192.stdout: mgr/telemetry/nag = false 2024-09-17T23:30:09.270 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.270 INFO:teuthology.orchestra.run.smithi192.stdout:[mon] 2024-09-17T23:30:09.270 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-09-17T23:30:09.270 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-09-17T23:30:09.270 INFO:teuthology.orchestra.run.smithi192.stdout: debug paxos = 20 2024-09-17T23:30:09.270 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-09-17T23:30:09.270 INFO:teuthology.orchestra.run.smithi192.stdout: mon data avail warn = 5 2024-09-17T23:30:09.270 INFO:teuthology.orchestra.run.smithi192.stdout: mon mgr mkfs grace = 240 2024-09-17T23:30:09.271 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:30:09.271 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd reporter subtree level = osd 2024-09-17T23:30:09.271 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd prime pg temp = true 2024-09-17T23:30:09.271 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:30:09.271 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.271 INFO:teuthology.orchestra.run.smithi192.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T23:30:09.271 INFO:teuthology.orchestra.run.smithi192.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T23:30:09.271 INFO:teuthology.orchestra.run.smithi192.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T23:30:09.271 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.271 INFO:teuthology.orchestra.run.smithi192.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T23:30:09.271 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T23:30:09.271 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T23:30:09.271 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.271 INFO:teuthology.orchestra.run.smithi192.stdout: # 1m isn't quite enough 2024-09-17T23:30:09.272 INFO:teuthology.orchestra.run.smithi192.stdout: mon_down_mkfs_grace = 2m 2024-09-17T23:30:09.272 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.272 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_filestore_osds = false 2024-09-17T23:30:09.272 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:09.272 INFO:teuthology.orchestra.run.smithi192.stdout:[client] 2024-09-17T23:30:09.272 INFO:teuthology.orchestra.run.smithi192.stdout: rgw cache enabled = true 2024-09-17T23:30:09.272 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable ops log = true 2024-09-17T23:30:09.272 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable usage log = true 2024-09-17T23:30:09.272 INFO:teuthology.orchestra.run.smithi192.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T23:30:09.272 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T23:30:09.272 INFO:teuthology.orchestra.run.smithi192.stdout: debug rbd = 20 2024-09-17T23:30:09.272 INFO:teuthology.orchestra.run.smithi192.stdout: debug rbd_mirror = 15 2024-09-17T23:30:09.272 INFO:teuthology.orchestra.run.smithi192.stdout: log to stderr = False 2024-09-17T23:30:09.273 INFO:teuthology.orchestra.run.smithi192.stdout: rbd default features = 125 2024-09-17T23:30:09.273 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.a] 2024-09-17T23:30:09.273 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.0] 2024-09-17T23:30:09.273 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:09.273 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:09.273 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.1] 2024-09-17T23:30:09.273 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:09.273 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:09.273 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.2] 2024-09-17T23:30:09.273 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:09.273 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:09.273 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.3] 2024-09-17T23:30:09.273 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:09.273 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:09.274 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.4] 2024-09-17T23:30:09.274 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:09.274 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:09.274 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.5] 2024-09-17T23:30:09.274 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:09.274 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:09.274 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.6] 2024-09-17T23:30:09.274 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:09.274 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:09.275 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-17T23:30:09.276 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-17T23:30:09.369 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-17T23:30:09.369 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:09.369 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-17T23:30:09.384 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:09.384 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-17T23:30:09.439 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi096.front.sepia.ceph.com 2024-09-17T23:30:09.439 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:09.439 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-17T23:30:09.439 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-17T23:30:09.513 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:09.513 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:30:09.568 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi192.front.sepia.ceph.com 2024-09-17T23:30:09.568 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:09.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-17T23:30:09.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-17T23:30:09.609 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:09.609 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:30:09.667 INFO:tasks.ceph:Setting up mon nodes... 2024-09-17T23:30:09.667 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-17T23:30:09.667 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-17T23:30:09.718 INFO:teuthology.orchestra.run.smithi096.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-17T23:30:09.720 INFO:tasks.ceph:Setting up mds nodes... 2024-09-17T23:30:09.720 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-17T23:30:09.721 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-17T23:30:09.761 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-17T23:30:09.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-17T23:30:09.830 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-17T23:30:09.845 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-17T23:30:09.888 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-17T23:30:09.902 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-17T23:30:09.943 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-17T23:30:09.956 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-17T23:30:10.037 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-17T23:30:10.050 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-17T23:30:10.131 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-17T23:30:10.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-17T23:30:10.224 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-17T23:30:10.237 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-17T23:30:10.318 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-17T23:30:10.331 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-17T23:30:10.411 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-17T23:30:10.424 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-17T23:30:10.424 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-17T23:30:10.424 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-17T23:30:10.450 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:30:10.451 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-17T23:30:10.451 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-17T23:30:10.451 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T23:30:10.524 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T23:30:10.524 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T23:30:10.524 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T23:30:10.525 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T23:30:10.525 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T23:30:10.525 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-09-17T23:30:10.525 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T23:30:10.525 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T23:30:10.525 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T23:30:10.525 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T23:30:11.029 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-09-17T23:30:11.032 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-09-17T23:30:11.032 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-17T23:30:11.195 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-17T23:30:11.231 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-17T23:30:11.299 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:30:11.299 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-17T23:30:11.299 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-17T23:30:11.299 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T23:30:11.373 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T23:30:11.374 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T23:30:11.374 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T23:30:11.374 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T23:30:11.374 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T23:30:11.374 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-09-17T23:30:11.374 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T23:30:11.374 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T23:30:11.374 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T23:30:11.374 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T23:30:11.819 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-09-17T23:30:11.821 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-09-17T23:30:11.821 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-17T23:30:11.916 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-17T23:30:11.945 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-17T23:30:12.011 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:30:12.011 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-17T23:30:12.011 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-17T23:30:12.011 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T23:30:12.084 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T23:30:12.085 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T23:30:12.085 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T23:30:12.085 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T23:30:12.085 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T23:30:12.085 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-09-17T23:30:12.086 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T23:30:12.086 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T23:30:12.086 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T23:30:12.086 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T23:30:12.577 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-09-17T23:30:12.578 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-09-17T23:30:12.579 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-17T23:30:12.706 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-17T23:30:12.735 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:30:12.833 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-17T23:30:12.831+0000 7f669c862640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-17T23:30:12.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-17T23:30:12.833+0000 7f669c862640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-17T23:30:12.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-17T23:30:12.833+0000 7f669c862640 -1 bdev(0x559c129bf800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2024-09-17T23:30:12.835 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-17T23:30:12.833+0000 7f669c862640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-17T23:30:16.256 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-17T23:30:16.287 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:30:16.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-17T23:30:16.376+0000 7f0c9ca0b640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-17T23:30:16.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-17T23:30:16.376+0000 7f0c9ca0b640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-17T23:30:16.378 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-17T23:30:16.376+0000 7f0c9ca0b640 -1 bdev(0x558c09043800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2024-09-17T23:30:16.379 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-17T23:30:16.376+0000 7f0c9ca0b640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-17T23:30:19.295 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-17T23:30:19.324 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:30:19.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-17T23:30:19.411+0000 7fe7f7012640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-17T23:30:19.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-17T23:30:19.411+0000 7fe7f7012640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-17T23:30:19.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-17T23:30:19.411+0000 7fe7f7012640 -1 bdev(0x5564a4cf9800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2024-09-17T23:30:19.413 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-17T23:30:19.411+0000 7fe7f7012640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-17T23:30:22.320 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-17T23:30:22.347 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-17T23:30:22.348 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:22.348 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-17T23:30:22.418 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:22.418 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-17T23:30:22.484 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:22.484 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-17T23:30:22.552 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:22.552 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-17T23:30:22.621 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:22.621 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-17T23:30:22.676 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:22.677 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-17T23:30:22.696 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:22.696 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-17T23:30:22.754 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:22.755 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-17T23:30:22.814 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:22.814 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-17T23:30:22.872 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:22.872 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-17T23:30:22.931 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:22.931 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-17T23:30:22.989 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:22.989 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-17T23:30:23.046 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:23.047 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-17T23:30:23.104 INFO:tasks.ceph:Adding keys to all mons... 2024-09-17T23:30:23.104 DEBUG:teuthology.orchestra.run.smithi096:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-17T23:30:23.128 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr.x] 2024-09-17T23:30:23.128 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQABEepmY1a8KhAAW3BdqxkLXDnXHRmdxapzDA== 2024-09-17T23:30:23.129 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.0] 2024-09-17T23:30:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQAEEepmji+eMRAAVOuT68B5NAdTZwqNdJEHxw== 2024-09-17T23:30:23.129 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.1] 2024-09-17T23:30:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQAIEepmTe54FhAADLb2tJCQD6G3HHJ9pDI+jg== 2024-09-17T23:30:23.129 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.2] 2024-09-17T23:30:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQALEepmjaKNGBAA0KDSZql42W4ZO8D5d1fQ2A== 2024-09-17T23:30:23.129 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-09-17T23:30:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQABEepm6o5QLRAAhPvdDZN3CFtzbcYjDzRa/A== 2024-09-17T23:30:23.129 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror] 2024-09-17T23:30:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQABEepmG5txMRAA93pq04i3VLDftm/Lien77g== 2024-09-17T23:30:23.129 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.0] 2024-09-17T23:30:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQABEepmolLoNBAAG2PVmEAEvWoGJOQtR4xwBQ== 2024-09-17T23:30:23.129 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.1] 2024-09-17T23:30:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQABEepmF8AuOBAAFS1C/BUJujmhMpjfGtgYCw== 2024-09-17T23:30:23.130 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.2] 2024-09-17T23:30:23.130 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQACEepmrwIyAhAAEJdAqu6LjU0l4mw7hxZxVw== 2024-09-17T23:30:23.130 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.3] 2024-09-17T23:30:23.130 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQACEepm16i+BxAAXqwkfmKM3gdXcz3R9pnKKg== 2024-09-17T23:30:23.130 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.4] 2024-09-17T23:30:23.130 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQACEepmGL9WDRAAjFpVnfdj7UbJe7JYT9BXAw== 2024-09-17T23:30:23.130 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.5] 2024-09-17T23:30:23.130 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQACEepmlOPnEhAA+3E7oaenLLIcoL0eom0FZg== 2024-09-17T23:30:23.130 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.6] 2024-09-17T23:30:23.130 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQACEepmpzh5GBAAph9fanNN6vZwfCedX95rRA== 2024-09-17T23:30:23.131 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T23:30:23.224 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T23:30:23.275 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T23:30:23.326 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T23:30:23.377 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:30:23.430 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:30:23.484 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:30:23.537 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:30:23.590 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:30:23.645 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:30:23.699 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:30:23.752 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:30:23.806 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:30:23.859 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-17T23:30:23.860 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-17T23:30:23.884 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-17T23:30:23.979 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-17T23:30:24.003 DEBUG:teuthology.orchestra.run.smithi096:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-17T23:30:24.059 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-17T23:30:24.060 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-17T23:30:24.060 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-17T23:30:24.102 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-17T23:30:24.102 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-17T23:30:24.103 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-17T23:30:24.103 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-17T23:30:24.105 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-17T23:30:24.105 INFO:tasks.ceph:Setting crush tunables to default 2024-09-17T23:30:24.105 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-17T23:30:24.276 INFO:teuthology.orchestra.run.smithi096.stderr:adjusted tunables profile to default 2024-09-17T23:30:24.288 INFO:tasks.ceph:check_enable_crimson: False 2024-09-17T23:30:24.288 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-17T23:30:24.288 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:24.288 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-17T23:30:24.313 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:24.313 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-17T23:30:24.378 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:24.378 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-17T23:30:24.443 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 osd new 19bbd5f7-1cb2-4cb0-ac9a-2f7c843445cb 0 2024-09-17T23:30:24.483 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:24.480+0000 7ff86e523040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T23:30:24.526 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:24.523+0000 7ff86e523040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T23:30:24.658 INFO:teuthology.orchestra.run.smithi096.stdout:0 2024-09-17T23:30:24.669 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 osd new 1ab00cad-2baf-4f29-88de-ca8a63a7ac41 1 2024-09-17T23:30:24.675 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:24.672+0000 7ff86e523040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T23:30:24.728 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:24.725+0000 7ff86e523040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T23:30:24.853 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-09-17T23:30:24.864 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 osd new 53a69aca-c21a-4d27-af52-f06da4af6119 2 2024-09-17T23:30:24.907 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:24.904+0000 7ff86e523040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T23:30:25.022 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:25.019+0000 7ff86e523040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T23:30:25.041 INFO:teuthology.orchestra.run.smithi096.stdout:2 2024-09-17T23:30:25.052 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-17T23:30:25.053 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-17T23:30:25.095 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-17T23:30:25.095 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-17T23:30:25.095 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-17T23:30:25.097 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-17T23:30:25.098 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-17T23:30:25.098 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-17T23:30:25.100 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-17T23:30:25.100 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-17T23:30:25.365 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:25.361+0000 7ff86e523040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T23:30:25.413 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:25.410+0000 7ff86e523040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T23:30:25.491 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:25.491 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":5,"fsid":"eeb17cc0-06d7-4ce4-845b-f0ddd9bde342","created":"2024-09-17T23:30:24.196667+0000","modified":"2024-09-17T23:30:25.039838+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T23:30:25.039838+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bbd5f7-1cb2-4cb0-ac9a-2f7c843445cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1ab00cad-2baf-4f29-88de-ca8a63a7ac41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"53a69aca-c21a-4d27-af52-f06da4af6119","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:30:25.502 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-17T23:30:25.502 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-17T23:30:25.568 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:25.564+0000 7ff86e523040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T23:30:25.805 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-17T23:30:25.968 INFO:teuthology.misc.health.smithi096.stdout: 2024-09-17T23:30:25.968 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":5,"fsid":"eeb17cc0-06d7-4ce4-845b-f0ddd9bde342","created":"2024-09-17T23:30:24.196667+0000","modified":"2024-09-17T23:30:25.039838+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T23:30:25.039838+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bbd5f7-1cb2-4cb0-ac9a-2f7c843445cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1ab00cad-2baf-4f29-88de-ca8a63a7ac41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"53a69aca-c21a-4d27-af52-f06da4af6119","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:30:25.978 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-17T23:30:26.325 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:30:26.322+0000 7f9c0b239640 -1 Falling back to public interface 2024-09-17T23:30:26.510 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:26.506+0000 7ff86e523040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T23:30:26.563 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:26.560+0000 7ff86e523040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T23:30:26.858 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:30:26.855+0000 7f707f064640 -1 Falling back to public interface 2024-09-17T23:30:26.931 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:26.928+0000 7ff86e523040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T23:30:27.016 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:27.012+0000 7ff86e523040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T23:30:27.068 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-17T23:30:27.065+0000 7f6ae034f640 -1 Falling back to public interface 2024-09-17T23:30:27.123 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:27.120+0000 7ff86e523040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T23:30:27.170 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:27.167+0000 7ff86e523040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T23:30:27.282 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:27.279+0000 7ff86e523040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T23:30:27.338 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:27.332+0000 7ff86e523040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T23:30:27.380 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:27.377+0000 7ff86e523040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T23:30:27.491 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:27.488+0000 7ff86e523040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T23:30:27.540 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:27.537+0000 7ff86e523040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T23:30:27.591 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:27.588+0000 7ff86e523040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T23:30:27.659 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:27.656+0000 7ff86e523040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T23:30:27.704 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:27.701+0000 7ff86e523040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T23:30:27.747 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:27.744+0000 7ff86e523040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T23:30:28.392 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:28.389+0000 7ff86e523040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T23:30:29.601 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-17T23:30:29.598+0000 7ff80e509640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-17T23:30:30.652 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:30:30.648+0000 7f9c0b239640 -1 osd.0 0 log_to_monitors true 2024-09-17T23:30:30.655 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:30:30.652+0000 7f707f064640 -1 osd.2 0 log_to_monitors true 2024-09-17T23:30:30.848 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-17T23:30:30.845+0000 7f6ae034f640 -1 osd.1 0 log_to_monitors true 2024-09-17T23:30:32.282 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-17T23:30:32.705 INFO:teuthology.misc.health.smithi096.stdout: 2024-09-17T23:30:32.705 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":7,"fsid":"eeb17cc0-06d7-4ce4-845b-f0ddd9bde342","created":"2024-09-17T23:30:24.196667+0000","modified":"2024-09-17T23:30:32.401996+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T23:30:25.039838+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"19bbd5f7-1cb2-4cb0-ac9a-2f7c843445cb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1ab00cad-2baf-4f29-88de-ca8a63a7ac41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"53a69aca-c21a-4d27-af52-f06da4af6119","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:30:32.722 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-17T23:30:33.430 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:30:33.427+0000 7f9c071c8640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T23:30:33.443 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-17T23:30:33.441+0000 7f6adc2de640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T23:30:33.444 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:30:33.442+0000 7f707b807640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T23:30:33.445 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:30:33.443+0000 7f9bfefe8640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T23:30:33.452 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-17T23:30:33.450+0000 7f6ad40fe640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T23:30:33.457 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:30:33.455+0000 7f7072e13640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T23:30:39.026 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-17T23:30:39.303 INFO:teuthology.misc.health.smithi096.stdout: 2024-09-17T23:30:39.303 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":12,"fsid":"eeb17cc0-06d7-4ce4-845b-f0ddd9bde342","created":"2024-09-17T23:30:24.196667+0000","modified":"2024-09-17T23:30:38.453452+0000","last_up_change":"2024-09-17T23:30:34.445001+0000","last_in_change":"2024-09-17T23:30:25.039838+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:30:36.425699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"19bbd5f7-1cb2-4cb0-ac9a-2f7c843445cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6801","nonce":3675985324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6803","nonce":3675985324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6807","nonce":3675985324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6805","nonce":3675985324}]},"public_addr":"172.21.15.96:6801/3675985324","cluster_addr":"172.21.15.96:6803/3675985324","heartbeat_back_addr":"172.21.15.96:6807/3675985324","heartbeat_front_addr":"172.21.15.96:6805/3675985324","state":["exists","up"]},{"osd":1,"uuid":"1ab00cad-2baf-4f29-88de-ca8a63a7ac41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6817","nonce":3450214342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6819","nonce":3450214342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6823","nonce":3450214342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6821","nonce":3450214342}]},"public_addr":"172.21.15.96:6817/3450214342","cluster_addr":"172.21.15.96:6819/3450214342","heartbeat_back_addr":"172.21.15.96:6823/3450214342","heartbeat_front_addr":"172.21.15.96:6821/3450214342","state":["exists","up"]},{"osd":2,"uuid":"53a69aca-c21a-4d27-af52-f06da4af6119","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6809","nonce":2203324171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6811","nonce":2203324171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6815","nonce":2203324171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6813","nonce":2203324171}]},"public_addr":"172.21.15.96:6809/2203324171","cluster_addr":"172.21.15.96:6811/2203324171","heartbeat_back_addr":"172.21.15.96:6815/2203324171","heartbeat_front_addr":"172.21.15.96:6813/2203324171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:30:31.673547+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:30:31.891181+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:30:31.700724+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:30:39.314 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-17T23:30:39.314 INFO:tasks.ceph:Creating RBD pool 2024-09-17T23:30:39.315 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-17T23:30:40.463 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'rbd' created 2024-09-17T23:30:40.480 DEBUG:teuthology.orchestra.run.smithi096:> rbd --cluster cluster1 pool init rbd 2024-09-17T23:30:43.485 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-17T23:30:43.486 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-17T23:30:43.486 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-17T23:30:43.486 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-17T23:30:43.486 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-17T23:30:43.820 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:43.832 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":497137675},{"type":"v1","addr":"172.21.15.96:6825","nonce":497137675}]},"active_addr":"172.21.15.96:6825/497137675","active_change":"2024-09-17T23:30:28.392588+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":319255789}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":3952241676}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":722233571}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":4188012180}]}]} 2024-09-17T23:30:43.835 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-17T23:30:43.836 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-17T23:30:43.836 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-17T23:30:44.120 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:44.120 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":16,"fsid":"eeb17cc0-06d7-4ce4-845b-f0ddd9bde342","created":"2024-09-17T23:30:24.196667+0000","modified":"2024-09-17T23:30:43.473541+0000","last_up_change":"2024-09-17T23:30:34.445001+0000","last_in_change":"2024-09-17T23:30:25.039838+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:30:36.425699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T23:30:39.605981+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"19bbd5f7-1cb2-4cb0-ac9a-2f7c843445cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6801","nonce":3675985324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6803","nonce":3675985324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6807","nonce":3675985324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6805","nonce":3675985324}]},"public_addr":"172.21.15.96:6801/3675985324","cluster_addr":"172.21.15.96:6803/3675985324","heartbeat_back_addr":"172.21.15.96:6807/3675985324","heartbeat_front_addr":"172.21.15.96:6805/3675985324","state":["exists","up"]},{"osd":1,"uuid":"1ab00cad-2baf-4f29-88de-ca8a63a7ac41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6817","nonce":3450214342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6819","nonce":3450214342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6823","nonce":3450214342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6821","nonce":3450214342}]},"public_addr":"172.21.15.96:6817/3450214342","cluster_addr":"172.21.15.96:6819/3450214342","heartbeat_back_addr":"172.21.15.96:6823/3450214342","heartbeat_front_addr":"172.21.15.96:6821/3450214342","state":["exists","up"]},{"osd":2,"uuid":"53a69aca-c21a-4d27-af52-f06da4af6119","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6809","nonce":2203324171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6811","nonce":2203324171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6815","nonce":2203324171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6813","nonce":2203324171}]},"public_addr":"172.21.15.96:6809/2203324171","cluster_addr":"172.21.15.96:6811/2203324171","heartbeat_back_addr":"172.21.15.96:6815/2203324171","heartbeat_front_addr":"172.21.15.96:6813/2203324171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:30:31.673547+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:30:31.891181+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:30:31.700724+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:30:44.133 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-17T23:30:44.133 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-17T23:30:44.429 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:44.429 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":16,"fsid":"eeb17cc0-06d7-4ce4-845b-f0ddd9bde342","created":"2024-09-17T23:30:24.196667+0000","modified":"2024-09-17T23:30:43.473541+0000","last_up_change":"2024-09-17T23:30:34.445001+0000","last_in_change":"2024-09-17T23:30:25.039838+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:30:36.425699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T23:30:39.605981+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"19bbd5f7-1cb2-4cb0-ac9a-2f7c843445cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6801","nonce":3675985324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6803","nonce":3675985324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6807","nonce":3675985324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6805","nonce":3675985324}]},"public_addr":"172.21.15.96:6801/3675985324","cluster_addr":"172.21.15.96:6803/3675985324","heartbeat_back_addr":"172.21.15.96:6807/3675985324","heartbeat_front_addr":"172.21.15.96:6805/3675985324","state":["exists","up"]},{"osd":1,"uuid":"1ab00cad-2baf-4f29-88de-ca8a63a7ac41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6817","nonce":3450214342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6819","nonce":3450214342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6823","nonce":3450214342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6821","nonce":3450214342}]},"public_addr":"172.21.15.96:6817/3450214342","cluster_addr":"172.21.15.96:6819/3450214342","heartbeat_back_addr":"172.21.15.96:6823/3450214342","heartbeat_front_addr":"172.21.15.96:6821/3450214342","state":["exists","up"]},{"osd":2,"uuid":"53a69aca-c21a-4d27-af52-f06da4af6119","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6809","nonce":2203324171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6811","nonce":2203324171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6815","nonce":2203324171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6813","nonce":2203324171}]},"public_addr":"172.21.15.96:6809/2203324171","cluster_addr":"172.21.15.96:6811/2203324171","heartbeat_back_addr":"172.21.15.96:6815/2203324171","heartbeat_front_addr":"172.21.15.96:6813/2203324171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:30:31.673547+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:30:31.891181+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:30:31.700724+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:30:44.443 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-17T23:30:44.443 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-17T23:30:44.444 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-17T23:30:44.598 INFO:teuthology.orchestra.run.smithi096.stdout:34359738372 2024-09-17T23:30:44.599 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-17T23:30:44.600 INFO:teuthology.orchestra.run.smithi096.stdout:34359738373 2024-09-17T23:30:44.600 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-17T23:30:44.601 INFO:teuthology.orchestra.run.smithi096.stdout:34359738372 2024-09-17T23:30:44.602 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-17T23:30:44.894 INFO:teuthology.orchestra.run.smithi096.stdout:34359738371 2024-09-17T23:30:44.913 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.1 2024-09-17T23:30:44.935 INFO:teuthology.orchestra.run.smithi096.stdout:34359738371 2024-09-17T23:30:44.941 INFO:teuthology.orchestra.run.smithi096.stdout:34359738372 2024-09-17T23:30:44.948 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.2 2024-09-17T23:30:44.954 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.0 2024-09-17T23:30:45.914 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-17T23:30:45.949 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-17T23:30:45.955 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-17T23:30:46.230 INFO:teuthology.orchestra.run.smithi096.stdout:34359738371 2024-09-17T23:30:46.244 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.1 2024-09-17T23:30:46.254 INFO:teuthology.orchestra.run.smithi096.stdout:34359738371 2024-09-17T23:30:46.261 INFO:teuthology.orchestra.run.smithi096.stdout:34359738372 2024-09-17T23:30:46.267 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.2 2024-09-17T23:30:46.274 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.0 2024-09-17T23:30:47.245 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-17T23:30:47.268 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-17T23:30:47.275 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-17T23:30:47.546 INFO:teuthology.orchestra.run.smithi096.stdout:34359738373 2024-09-17T23:30:47.563 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738373 for osd.1 2024-09-17T23:30:47.563 DEBUG:teuthology.parallel:result is None 2024-09-17T23:30:47.584 INFO:teuthology.orchestra.run.smithi096.stdout:34359738374 2024-09-17T23:30:47.596 INFO:teuthology.orchestra.run.smithi096.stdout:34359738373 2024-09-17T23:30:47.596 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738374 for osd.0 2024-09-17T23:30:47.597 DEBUG:teuthology.parallel:result is None 2024-09-17T23:30:47.608 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738373 for osd.2 2024-09-17T23:30:47.609 DEBUG:teuthology.parallel:result is None 2024-09-17T23:30:47.609 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-17T23:30:47.609 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-17T23:30:47.918 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:47.918 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-17T23:30:47.930 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-17T23:30:46.398563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81624,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033896,"statfs":{"total":289910292480,"available":289826709504,"internally_reserved":0,"allocated":540672,"data_stored":1303162,"data_compressed":5782,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.936475"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.479739+0000","last_change":"2024-09-17T23:30:43.479739+0000","last_active":"2024-09-17T23:30:43.479739+0000","last_peered":"2024-09-17T23:30:43.479739+0000","last_clean":"2024-09-17T23:30:43.479739+0000","last_became_active":"2024-09-17T23:30:41.472305+0000","last_became_peered":"2024-09-17T23:30:41.472305+0000","last_unstale":"2024-09-17T23:30:43.479739+0000","last_undegraded":"2024-09-17T23:30:43.479739+0000","last_fullsized":"2024-09-17T23:30:43.479739+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:14:19.225045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050569899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.480063+0000","last_change":"2024-09-17T23:30:43.480063+0000","last_active":"2024-09-17T23:30:43.480063+0000","last_peered":"2024-09-17T23:30:43.480063+0000","last_clean":"2024-09-17T23:30:43.480063+0000","last_became_active":"2024-09-17T23:30:41.471899+0000","last_became_peered":"2024-09-17T23:30:41.471899+0000","last_unstale":"2024-09-17T23:30:43.480063+0000","last_undegraded":"2024-09-17T23:30:43.480063+0000","last_fullsized":"2024-09-17T23:30:43.480063+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:25:46.122761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082020499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.480349+0000","last_change":"2024-09-17T23:30:43.480349+0000","last_active":"2024-09-17T23:30:43.480349+0000","last_peered":"2024-09-17T23:30:43.480349+0000","last_clean":"2024-09-17T23:30:43.480349+0000","last_became_active":"2024-09-17T23:30:41.471801+0000","last_became_peered":"2024-09-17T23:30:41.471801+0000","last_unstale":"2024-09-17T23:30:43.480349+0000","last_undegraded":"2024-09-17T23:30:43.480349+0000","last_fullsized":"2024-09-17T23:30:43.480349+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:17:10.596752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00106452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.480110+0000","last_change":"2024-09-17T23:30:43.480110+0000","last_active":"2024-09-17T23:30:43.480110+0000","last_peered":"2024-09-17T23:30:43.480110+0000","last_clean":"2024-09-17T23:30:43.480110+0000","last_became_active":"2024-09-17T23:30:41.471678+0000","last_became_peered":"2024-09-17T23:30:41.471678+0000","last_unstale":"2024-09-17T23:30:43.480110+0000","last_undegraded":"2024-09-17T23:30:43.480110+0000","last_fullsized":"2024-09-17T23:30:43.480110+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:16:48.723032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088593199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.478589+0000","last_change":"2024-09-17T23:30:43.476525+0000","last_active":"2024-09-17T23:30:43.478589+0000","last_peered":"2024-09-17T23:30:43.478589+0000","last_clean":"2024-09-17T23:30:43.478589+0000","last_became_active":"2024-09-17T23:30:41.470241+0000","last_became_peered":"2024-09-17T23:30:41.470241+0000","last_unstale":"2024-09-17T23:30:43.478589+0000","last_undegraded":"2024-09-17T23:30:43.478589+0000","last_fullsized":"2024-09-17T23:30:43.478589+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:48:45.144310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051423599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.635997+0000","last_change":"2024-09-17T23:30:43.635997+0000","last_active":"2024-09-17T23:30:43.635997+0000","last_peered":"2024-09-17T23:30:43.635997+0000","last_clean":"2024-09-17T23:30:43.635997+0000","last_became_active":"2024-09-17T23:30:41.474920+0000","last_became_peered":"2024-09-17T23:30:41.474920+0000","last_unstale":"2024-09-17T23:30:43.635997+0000","last_undegraded":"2024-09-17T23:30:43.635997+0000","last_fullsized":"2024-09-17T23:30:43.635997+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:19:55.884592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041385499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.636000+0000","last_change":"2024-09-17T23:30:43.636000+0000","last_active":"2024-09-17T23:30:43.636000+0000","last_peered":"2024-09-17T23:30:43.636000+0000","last_clean":"2024-09-17T23:30:43.636000+0000","last_became_active":"2024-09-17T23:30:41.474871+0000","last_became_peered":"2024-09-17T23:30:41.474871+0000","last_unstale":"2024-09-17T23:30:43.636000+0000","last_undegraded":"2024-09-17T23:30:43.636000+0000","last_fullsized":"2024-09-17T23:30:43.636000+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:06:22.909014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041168599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.479732+0000","last_change":"2024-09-17T23:30:43.479732+0000","last_active":"2024-09-17T23:30:43.479732+0000","last_peered":"2024-09-17T23:30:43.479732+0000","last_clean":"2024-09-17T23:30:43.479732+0000","last_became_active":"2024-09-17T23:30:41.473769+0000","last_became_peered":"2024-09-17T23:30:41.473769+0000","last_unstale":"2024-09-17T23:30:43.479732+0000","last_undegraded":"2024-09-17T23:30:43.479732+0000","last_fullsized":"2024-09-17T23:30:43.479732+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:09:57.242769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054243800000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":146,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.479362+0000","last_change":"2024-09-17T23:30:37.576035+0000","last_active":"2024-09-17T23:30:43.479362+0000","last_peered":"2024-09-17T23:30:43.479362+0000","last_clean":"2024-09-17T23:30:43.479362+0000","last_became_active":"2024-09-17T23:30:37.575069+0000","last_became_peered":"2024-09-17T23:30:37.575069+0000","last_unstale":"2024-09-17T23:30:43.479362+0000","last_undegraded":"2024-09-17T23:30:43.479362+0000","last_fullsized":"2024-09-17T23:30:43.479362+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:36.447566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:36.447566+0000","last_clean_scrub_stamp":"2024-09-17T23:30:36.447566+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:27:48.840294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5782,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":139264,"data_stored":40796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":2891,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":2891,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2891,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2891,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:30:47.931 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-17T23:30:48.202 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:48.203 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-17T23:30:48.215 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-17T23:30:46.398563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81624,"kb_used_data":528,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033896,"statfs":{"total":289910292480,"available":289826709504,"internally_reserved":0,"allocated":540672,"data_stored":1303162,"data_compressed":5782,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.936475"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.479739+0000","last_change":"2024-09-17T23:30:43.479739+0000","last_active":"2024-09-17T23:30:43.479739+0000","last_peered":"2024-09-17T23:30:43.479739+0000","last_clean":"2024-09-17T23:30:43.479739+0000","last_became_active":"2024-09-17T23:30:41.472305+0000","last_became_peered":"2024-09-17T23:30:41.472305+0000","last_unstale":"2024-09-17T23:30:43.479739+0000","last_undegraded":"2024-09-17T23:30:43.479739+0000","last_fullsized":"2024-09-17T23:30:43.479739+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:14:19.225045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050569899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.480063+0000","last_change":"2024-09-17T23:30:43.480063+0000","last_active":"2024-09-17T23:30:43.480063+0000","last_peered":"2024-09-17T23:30:43.480063+0000","last_clean":"2024-09-17T23:30:43.480063+0000","last_became_active":"2024-09-17T23:30:41.471899+0000","last_became_peered":"2024-09-17T23:30:41.471899+0000","last_unstale":"2024-09-17T23:30:43.480063+0000","last_undegraded":"2024-09-17T23:30:43.480063+0000","last_fullsized":"2024-09-17T23:30:43.480063+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:25:46.122761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082020499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.480349+0000","last_change":"2024-09-17T23:30:43.480349+0000","last_active":"2024-09-17T23:30:43.480349+0000","last_peered":"2024-09-17T23:30:43.480349+0000","last_clean":"2024-09-17T23:30:43.480349+0000","last_became_active":"2024-09-17T23:30:41.471801+0000","last_became_peered":"2024-09-17T23:30:41.471801+0000","last_unstale":"2024-09-17T23:30:43.480349+0000","last_undegraded":"2024-09-17T23:30:43.480349+0000","last_fullsized":"2024-09-17T23:30:43.480349+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:17:10.596752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00106452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.480110+0000","last_change":"2024-09-17T23:30:43.480110+0000","last_active":"2024-09-17T23:30:43.480110+0000","last_peered":"2024-09-17T23:30:43.480110+0000","last_clean":"2024-09-17T23:30:43.480110+0000","last_became_active":"2024-09-17T23:30:41.471678+0000","last_became_peered":"2024-09-17T23:30:41.471678+0000","last_unstale":"2024-09-17T23:30:43.480110+0000","last_undegraded":"2024-09-17T23:30:43.480110+0000","last_fullsized":"2024-09-17T23:30:43.480110+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:16:48.723032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088593199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":21,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.478589+0000","last_change":"2024-09-17T23:30:43.476525+0000","last_active":"2024-09-17T23:30:43.478589+0000","last_peered":"2024-09-17T23:30:43.478589+0000","last_clean":"2024-09-17T23:30:43.478589+0000","last_became_active":"2024-09-17T23:30:41.470241+0000","last_became_peered":"2024-09-17T23:30:41.470241+0000","last_unstale":"2024-09-17T23:30:43.478589+0000","last_undegraded":"2024-09-17T23:30:43.478589+0000","last_fullsized":"2024-09-17T23:30:43.478589+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:48:45.144310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051423599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.635997+0000","last_change":"2024-09-17T23:30:43.635997+0000","last_active":"2024-09-17T23:30:43.635997+0000","last_peered":"2024-09-17T23:30:43.635997+0000","last_clean":"2024-09-17T23:30:43.635997+0000","last_became_active":"2024-09-17T23:30:41.474920+0000","last_became_peered":"2024-09-17T23:30:41.474920+0000","last_unstale":"2024-09-17T23:30:43.635997+0000","last_undegraded":"2024-09-17T23:30:43.635997+0000","last_fullsized":"2024-09-17T23:30:43.635997+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:19:55.884592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041385499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.636000+0000","last_change":"2024-09-17T23:30:43.636000+0000","last_active":"2024-09-17T23:30:43.636000+0000","last_peered":"2024-09-17T23:30:43.636000+0000","last_clean":"2024-09-17T23:30:43.636000+0000","last_became_active":"2024-09-17T23:30:41.474871+0000","last_became_peered":"2024-09-17T23:30:41.474871+0000","last_unstale":"2024-09-17T23:30:43.636000+0000","last_undegraded":"2024-09-17T23:30:43.636000+0000","last_fullsized":"2024-09-17T23:30:43.636000+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:06:22.909014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041168599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.479732+0000","last_change":"2024-09-17T23:30:43.479732+0000","last_active":"2024-09-17T23:30:43.479732+0000","last_peered":"2024-09-17T23:30:43.479732+0000","last_clean":"2024-09-17T23:30:43.479732+0000","last_became_active":"2024-09-17T23:30:41.473769+0000","last_became_peered":"2024-09-17T23:30:41.473769+0000","last_unstale":"2024-09-17T23:30:43.479732+0000","last_undegraded":"2024-09-17T23:30:43.479732+0000","last_fullsized":"2024-09-17T23:30:43.479732+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:09:57.242769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054243800000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":146,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-17T23:30:43.479362+0000","last_change":"2024-09-17T23:30:37.576035+0000","last_active":"2024-09-17T23:30:43.479362+0000","last_peered":"2024-09-17T23:30:43.479362+0000","last_clean":"2024-09-17T23:30:43.479362+0000","last_became_active":"2024-09-17T23:30:37.575069+0000","last_became_peered":"2024-09-17T23:30:37.575069+0000","last_unstale":"2024-09-17T23:30:43.479362+0000","last_undegraded":"2024-09-17T23:30:43.479362+0000","last_fullsized":"2024-09-17T23:30:43.479362+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:36.447566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:36.447566+0000","last_clean_scrub_stamp":"2024-09-17T23:30:36.447566+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:27:48.840294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5782,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":139264,"data_stored":40796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":2891,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":200704,"data_stored":631183,"data_compressed":2891,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2891,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2891,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:30:48.216 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-17T23:30:48.216 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-17T23:30:48.216 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-17T23:30:48.216 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-17T23:30:48.526 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:48.526 INFO:teuthology.orchestra.run.smithi096.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T23:30:48.538 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-17T23:30:48.538 INFO:teuthology.run_tasks:Running task ceph... 2024-09-17T23:30:48.548 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-17T23:30:48.548 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-17T23:30:48.548 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909591', 'branch': 'squid', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_latest} workloads/compare-mirror-images-krbd}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7909591', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}], 'name': 'teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}, 'extra_system_packages': ['pv']}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'b0d356b98e08f6bc59ef15652f0d0e4298fc83c7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 1983, 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f', 'sleep_before_teardown': 0, 'subset': '79/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_b0d356b98e08f6bc59ef15652f0d0e4298fc83c7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'b0d356b98e08f6bc59ef15652f0d0e4298fc83c7', 'targets': {'smithi096.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIvRkATk5dJFdg8bZ75kCPOVVti6XPuOAeNrFoSrWb1IZV+uOrgL1tk+67NoeOcH0c7OiLJ0627Yp39IrVU+X+A=', 'smithi192.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB07Wg4TIz69AdzDYYYnJEbgHMOwtweLxwBPUjQITs/NiMWDkBh0Ms3D/BGVdlo9PZk+OtslZcKDf/1fgkLbihA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f', 'extra_system_packages': ['pv']}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/compare_mirror_images.sh']}, 'env': {'RBD_DEVICE_TYPE': 'krbd'}, 'timeout': '3h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-17T23:30:48.549 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-17T23:30:48.551 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-17T23:30:48.571 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T23:30:48.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/run/ceph 2024-09-17T23:30:48.641 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:48.641 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T23:30:48.700 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T23:30:48.700 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-09-17T23:30:48.758 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T23:30:48.758 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:30:48.758 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-17T23:30:48.758 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:30:48.758 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:30:48.759 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-09-17 23:29:59.126832133 +0000 2024-09-17T23:30:48.759 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-09-17 23:28:08.548913177 +0000 2024-09-17T23:30:48.759 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-09-17 23:28:08.548913177 +0000 2024-09-17T23:30:48.759 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-09-17 23:28:08.548913177 +0000 2024-09-17T23:30:48.759 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T23:30:48.826 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-09-17T23:30:48.826 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-09-17T23:30:48.826 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000196146 s, 2.6 MB/s 2024-09-17T23:30:48.827 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T23:30:48.886 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-09-17T23:30:48.944 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T23:30:48.944 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:30:48.944 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-17T23:30:48.944 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:30:48.944 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:30:48.944 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-09-17 23:29:59.126832133 +0000 2024-09-17T23:30:48.944 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-09-17 23:28:08.874907004 +0000 2024-09-17T23:30:48.944 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-09-17 23:28:08.874907004 +0000 2024-09-17T23:30:48.945 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-09-17 23:28:08.874907004 +0000 2024-09-17T23:30:48.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T23:30:49.009 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-09-17T23:30:49.010 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-09-17T23:30:49.010 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000211082 s, 2.4 MB/s 2024-09-17T23:30:49.011 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T23:30:49.068 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-09-17T23:30:49.124 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T23:30:49.124 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:30:49.125 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-17T23:30:49.125 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:30:49.125 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:30:49.125 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-09-17 23:29:59.126832133 +0000 2024-09-17T23:30:49.125 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-09-17 23:28:09.192900984 +0000 2024-09-17T23:30:49.125 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-09-17 23:28:09.192900984 +0000 2024-09-17T23:30:49.125 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-09-17 23:28:09.192900984 +0000 2024-09-17T23:30:49.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T23:30:49.189 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-09-17T23:30:49.189 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-09-17T23:30:49.189 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000213625 s, 2.4 MB/s 2024-09-17T23:30:49.190 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T23:30:49.248 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-09-17T23:30:49.303 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T23:30:49.304 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T23:30:49.304 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-17T23:30:49.304 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T23:30:49.304 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T23:30:49.304 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-09-17 23:29:59.127832113 +0000 2024-09-17T23:30:49.304 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-09-17 23:28:09.518894813 +0000 2024-09-17T23:30:49.304 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-09-17 23:28:09.518894813 +0000 2024-09-17T23:30:49.304 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-09-17 23:28:09.518894813 +0000 2024-09-17T23:30:49.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T23:30:49.369 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-09-17T23:30:49.369 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-09-17T23:30:49.369 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000158674 s, 3.2 MB/s 2024-09-17T23:30:49.370 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T23:30:49.425 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:30:49.425 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-17T23:30:49.425 INFO:tasks.ceph:Generating config... 2024-09-17T23:30:49.426 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-17T23:30:49.426 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-17T23:30:49.427 INFO:tasks.ceph:[client] log to stderr = False 2024-09-17T23:30:49.427 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-17T23:30:49.427 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.427 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.427 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.427 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.427 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.427 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.427 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.427 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.427 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.427 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.427 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.428 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.428 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.428 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.428 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-17T23:30:49.428 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-17T23:30:49.428 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-17T23:30:49.428 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-17T23:30:49.428 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-17T23:30:49.428 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-17T23:30:49.428 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-17T23:30:49.428 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-17T23:30:49.428 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-17T23:30:49.428 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-09-17T23:30:49.429 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-17T23:30:49.429 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-17T23:30:49.429 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-17T23:30:49.429 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-17T23:30:49.429 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-17T23:30:49.429 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-17T23:30:49.429 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-17T23:30:49.429 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-17T23:30:49.429 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-17T23:30:49.429 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-17T23:30:49.429 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-17T23:30:49.429 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-17T23:30:49.430 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-17T23:30:49.430 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-17T23:30:49.505 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.keyring 2024-09-17T23:30:49.508 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-17T23:30:49.591 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-17T23:30:49.656 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.192')] 2024-09-17T23:30:49.656 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.192', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-17T23:30:49.657 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:49.657 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T23:30:49.711 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.192 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:30:49.791 INFO:teuthology.orchestra.run.smithi192.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:30:49.791 INFO:teuthology.orchestra.run.smithi192.stdout:monmaptool: generated fsid d61484e8-a8cb-4030-b6c2-cef28a296bff 2024-09-17T23:30:49.791 INFO:teuthology.orchestra.run.smithi192.stdout:setting min_mon_release = quincy 2024-09-17T23:30:49.791 INFO:teuthology.orchestra.run.smithi192.stdout:epoch 0 2024-09-17T23:30:49.791 INFO:teuthology.orchestra.run.smithi192.stdout:fsid d61484e8-a8cb-4030-b6c2-cef28a296bff 2024-09-17T23:30:49.792 INFO:teuthology.orchestra.run.smithi192.stdout:last_changed 2024-09-17T23:30:49.790050+0000 2024-09-17T23:30:49.792 INFO:teuthology.orchestra.run.smithi192.stdout:created 2024-09-17T23:30:49.790050+0000 2024-09-17T23:30:49.792 INFO:teuthology.orchestra.run.smithi192.stdout:min_mon_release 17 (quincy) 2024-09-17T23:30:49.792 INFO:teuthology.orchestra.run.smithi192.stdout:election_strategy: 1 2024-09-17T23:30:49.792 INFO:teuthology.orchestra.run.smithi192.stdout:0: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.a 2024-09-17T23:30:49.792 INFO:teuthology.orchestra.run.smithi192.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-17T23:30:49.793 DEBUG:teuthology.orchestra.run.smithi192:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-17T23:30:49.848 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID d61484e8-a8cb-4030-b6c2-cef28a296bff... 2024-09-17T23:30:49.849 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-17T23:30:49.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-17T23:30:49.900 INFO:teuthology.orchestra.run.smithi096.stdout:[global] 2024-09-17T23:30:49.901 INFO:teuthology.orchestra.run.smithi096.stdout: chdir = "" 2024-09-17T23:30:49.901 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T23:30:49.901 INFO:teuthology.orchestra.run.smithi096.stdout: auth supported = cephx 2024-09-17T23:30:49.901 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.901 INFO:teuthology.orchestra.run.smithi096.stdout: filestore xattr use omap = true 2024-09-17T23:30:49.901 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.901 INFO:teuthology.orchestra.run.smithi096.stdout: mon clock drift allowed = 1.000 2024-09-17T23:30:49.901 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.901 INFO:teuthology.orchestra.run.smithi096.stdout: osd crush chooseleaf type = 0 2024-09-17T23:30:49.901 INFO:teuthology.orchestra.run.smithi096.stdout: auth debug = true 2024-09-17T23:30:49.901 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.902 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on old message = true 2024-09-17T23:30:49.902 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on bug = true 2024-09-17T23:30:49.902 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.902 INFO:teuthology.orchestra.run.smithi096.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T23:30:49.902 INFO:teuthology.orchestra.run.smithi096.stdout: mon pg warn max object skew = 0 2024-09-17T23:30:49.902 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.902 INFO:teuthology.orchestra.run.smithi096.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T23:30:49.902 INFO:teuthology.orchestra.run.smithi096.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T23:30:49.902 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.902 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default size = 2 2024-09-17T23:30:49.902 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.902 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow primary affinity = true 2024-09-17T23:30:49.902 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow pg remap = true 2024-09-17T23:30:49.902 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on legacy crush tunables = false 2024-09-17T23:30:49.903 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on crush straw calc version zero = false 2024-09-17T23:30:49.903 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on no sortbitwise = false 2024-09-17T23:30:49.903 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on osd down out interval zero = false 2024-09-17T23:30:49.903 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on too few osds = false 2024-09-17T23:30:49.903 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T23:30:49.903 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T23:30:49.903 INFO:teuthology.orchestra.run.smithi096.stdout: mon_allow_pool_size_one = true 2024-09-17T23:30:49.903 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.903 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T23:30:49.903 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.903 INFO:teuthology.orchestra.run.smithi096.stdout: osd default data pool replay window = 5 2024-09-17T23:30:49.903 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.903 INFO:teuthology.orchestra.run.smithi096.stdout: mon allow pool delete = true 2024-09-17T23:30:49.904 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.904 INFO:teuthology.orchestra.run.smithi096.stdout: mon cluster log file level = debug 2024-09-17T23:30:49.904 INFO:teuthology.orchestra.run.smithi096.stdout: debug asserts on shutdown = true 2024-09-17T23:30:49.904 INFO:teuthology.orchestra.run.smithi096.stdout: mon health detail to clog = false 2024-09-17T23:30:49.904 INFO:teuthology.orchestra.run.smithi096.stdout: mon host = 172.21.15.192 2024-09-17T23:30:49.904 INFO:teuthology.orchestra.run.smithi096.stdout: mon client directed command retry = 5 2024-09-17T23:30:49.904 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on pool no app = False 2024-09-17T23:30:49.904 INFO:teuthology.orchestra.run.smithi096.stdout: ms inject socket failures = 5000 2024-09-17T23:30:49.904 INFO:teuthology.orchestra.run.smithi096.stdout: fsid = d61484e8-a8cb-4030-b6c2-cef28a296bff 2024-09-17T23:30:49.904 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.904 INFO:teuthology.orchestra.run.smithi096.stdout:[osd] 2024-09-17T23:30:49.904 INFO:teuthology.orchestra.run.smithi096.stdout: osd journal size = 100 2024-09-17T23:30:49.904 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.904 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub load threshold = 5.0 2024-09-17T23:30:49.905 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub max interval = 600 2024-09-17T23:30:49.905 INFO:teuthology.orchestra.run.smithi096.stdout: osd mclock profile = high_recovery_ops 2024-09-17T23:30:49.905 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.905 INFO:teuthology.orchestra.run.smithi096.stdout: osd recover clone overlap = true 2024-09-17T23:30:49.905 INFO:teuthology.orchestra.run.smithi096.stdout: osd recovery max chunk = 1048576 2024-09-17T23:30:49.905 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.905 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug shutdown = true 2024-09-17T23:30:49.905 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug op order = true 2024-09-17T23:30:49.905 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify stray on activate = true 2024-09-17T23:30:49.905 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug trim objects = true 2024-09-17T23:30:49.905 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.906 INFO:teuthology.orchestra.run.smithi096.stdout: osd open classes on start = true 2024-09-17T23:30:49.906 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug pg log writeout = true 2024-09-17T23:30:49.906 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.906 INFO:teuthology.orchestra.run.smithi096.stdout: osd deep scrub update digest min age = 30 2024-09-17T23:30:49.906 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.906 INFO:teuthology.orchestra.run.smithi096.stdout: osd map max advance = 10 2024-09-17T23:30:49.906 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.906 INFO:teuthology.orchestra.run.smithi096.stdout: journal zero on create = true 2024-09-17T23:30:49.906 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.906 INFO:teuthology.orchestra.run.smithi096.stdout: filestore ondisk finisher threads = 3 2024-09-17T23:30:49.906 INFO:teuthology.orchestra.run.smithi096.stdout: filestore apply finisher threads = 3 2024-09-17T23:30:49.906 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.906 INFO:teuthology.orchestra.run.smithi096.stdout: bdev debug aio = true 2024-09-17T23:30:49.906 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug misdirected ops = true 2024-09-17T23:30:49.907 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore block size = 96636764160 2024-09-17T23:30:49.907 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore compression algorithm = zstd 2024-09-17T23:30:49.907 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore compression mode = aggressive 2024-09-17T23:30:49.907 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore fsck on mount = True 2024-09-17T23:30:49.907 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluefs = 1/20 2024-09-17T23:30:49.907 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluestore = 1/20 2024-09-17T23:30:49.907 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-17T23:30:49.907 INFO:teuthology.orchestra.run.smithi096.stdout: debug osd = 20 2024-09-17T23:30:49.907 INFO:teuthology.orchestra.run.smithi096.stdout: debug rocksdb = 4/10 2024-09-17T23:30:49.907 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T23:30:49.907 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd full ratio = 0.9 2024-09-17T23:30:49.907 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd nearfull ratio = 0.8 2024-09-17T23:30:49.907 INFO:teuthology.orchestra.run.smithi096.stdout: osd failsafe full ratio = 0.95 2024-09-17T23:30:49.907 INFO:teuthology.orchestra.run.smithi096.stdout: osd objectstore = bluestore 2024-09-17T23:30:49.908 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.908 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr] 2024-09-17T23:30:49.908 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-17T23:30:49.908 INFO:teuthology.orchestra.run.smithi096.stdout: debug mgr = 20 2024-09-17T23:30:49.908 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-09-17T23:30:49.908 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-09-17T23:30:49.908 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:30:49.908 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:30:49.908 INFO:teuthology.orchestra.run.smithi096.stdout: mgr/telemetry/nag = false 2024-09-17T23:30:49.908 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.908 INFO:teuthology.orchestra.run.smithi096.stdout:[mon] 2024-09-17T23:30:49.908 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-17T23:30:49.908 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-09-17T23:30:49.909 INFO:teuthology.orchestra.run.smithi096.stdout: debug paxos = 20 2024-09-17T23:30:49.909 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-09-17T23:30:49.909 INFO:teuthology.orchestra.run.smithi096.stdout: mon data avail warn = 5 2024-09-17T23:30:49.909 INFO:teuthology.orchestra.run.smithi096.stdout: mon mgr mkfs grace = 240 2024-09-17T23:30:49.909 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:30:49.909 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd reporter subtree level = osd 2024-09-17T23:30:49.909 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd prime pg temp = true 2024-09-17T23:30:49.909 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:30:49.909 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.909 INFO:teuthology.orchestra.run.smithi096.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T23:30:49.909 INFO:teuthology.orchestra.run.smithi096.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T23:30:49.909 INFO:teuthology.orchestra.run.smithi096.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T23:30:49.909 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.909 INFO:teuthology.orchestra.run.smithi096.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T23:30:49.910 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T23:30:49.910 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T23:30:49.910 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.910 INFO:teuthology.orchestra.run.smithi096.stdout: # 1m isn't quite enough 2024-09-17T23:30:49.910 INFO:teuthology.orchestra.run.smithi096.stdout: mon_down_mkfs_grace = 2m 2024-09-17T23:30:49.910 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.910 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_filestore_osds = false 2024-09-17T23:30:49.910 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-17T23:30:49.910 INFO:teuthology.orchestra.run.smithi096.stdout:[client] 2024-09-17T23:30:49.911 INFO:teuthology.orchestra.run.smithi096.stdout: rgw cache enabled = true 2024-09-17T23:30:49.911 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable ops log = true 2024-09-17T23:30:49.911 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable usage log = true 2024-09-17T23:30:49.911 INFO:teuthology.orchestra.run.smithi096.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T23:30:49.911 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T23:30:49.911 INFO:teuthology.orchestra.run.smithi096.stdout: debug rbd = 20 2024-09-17T23:30:49.911 INFO:teuthology.orchestra.run.smithi096.stdout: debug rbd_mirror = 15 2024-09-17T23:30:49.911 INFO:teuthology.orchestra.run.smithi096.stdout: log to stderr = False 2024-09-17T23:30:49.911 INFO:teuthology.orchestra.run.smithi096.stdout: rbd default features = 125 2024-09-17T23:30:49.911 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.a] 2024-09-17T23:30:49.911 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.0] 2024-09-17T23:30:49.911 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.911 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.912 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.1] 2024-09-17T23:30:49.912 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.912 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.912 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.2] 2024-09-17T23:30:49.912 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.912 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.912 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.3] 2024-09-17T23:30:49.912 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.912 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.912 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.4] 2024-09-17T23:30:49.912 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.912 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.912 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.5] 2024-09-17T23:30:49.912 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.913 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.913 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.6] 2024-09-17T23:30:49.913 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.913 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.930 INFO:teuthology.orchestra.run.smithi192.stdout:[global] 2024-09-17T23:30:49.930 INFO:teuthology.orchestra.run.smithi192.stdout: chdir = "" 2024-09-17T23:30:49.930 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-17T23:30:49.931 INFO:teuthology.orchestra.run.smithi192.stdout: auth supported = cephx 2024-09-17T23:30:49.931 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.931 INFO:teuthology.orchestra.run.smithi192.stdout: filestore xattr use omap = true 2024-09-17T23:30:49.931 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.931 INFO:teuthology.orchestra.run.smithi192.stdout: mon clock drift allowed = 1.000 2024-09-17T23:30:49.931 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.931 INFO:teuthology.orchestra.run.smithi192.stdout: osd crush chooseleaf type = 0 2024-09-17T23:30:49.931 INFO:teuthology.orchestra.run.smithi192.stdout: auth debug = true 2024-09-17T23:30:49.931 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.931 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on old message = true 2024-09-17T23:30:49.931 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on bug = true 2024-09-17T23:30:49.931 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.931 INFO:teuthology.orchestra.run.smithi192.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-17T23:30:49.932 INFO:teuthology.orchestra.run.smithi192.stdout: mon pg warn max object skew = 0 2024-09-17T23:30:49.932 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.932 INFO:teuthology.orchestra.run.smithi192.stdout: # disable pg_autoscaler by default for new pools 2024-09-17T23:30:49.932 INFO:teuthology.orchestra.run.smithi192.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-17T23:30:49.932 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.932 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default size = 2 2024-09-17T23:30:49.932 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.932 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow primary affinity = true 2024-09-17T23:30:49.932 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow pg remap = true 2024-09-17T23:30:49.932 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on legacy crush tunables = false 2024-09-17T23:30:49.932 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on crush straw calc version zero = false 2024-09-17T23:30:49.932 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on no sortbitwise = false 2024-09-17T23:30:49.932 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on osd down out interval zero = false 2024-09-17T23:30:49.933 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on too few osds = false 2024-09-17T23:30:49.933 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-17T23:30:49.933 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-17T23:30:49.933 INFO:teuthology.orchestra.run.smithi192.stdout: mon_allow_pool_size_one = true 2024-09-17T23:30:49.933 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.933 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-17T23:30:49.933 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.933 INFO:teuthology.orchestra.run.smithi192.stdout: osd default data pool replay window = 5 2024-09-17T23:30:49.933 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.933 INFO:teuthology.orchestra.run.smithi192.stdout: mon allow pool delete = true 2024-09-17T23:30:49.933 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.933 INFO:teuthology.orchestra.run.smithi192.stdout: mon cluster log file level = debug 2024-09-17T23:30:49.934 INFO:teuthology.orchestra.run.smithi192.stdout: debug asserts on shutdown = true 2024-09-17T23:30:49.934 INFO:teuthology.orchestra.run.smithi192.stdout: mon health detail to clog = false 2024-09-17T23:30:49.934 INFO:teuthology.orchestra.run.smithi192.stdout: mon host = 172.21.15.192 2024-09-17T23:30:49.934 INFO:teuthology.orchestra.run.smithi192.stdout: mon client directed command retry = 5 2024-09-17T23:30:49.934 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on pool no app = False 2024-09-17T23:30:49.934 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject socket failures = 5000 2024-09-17T23:30:49.934 INFO:teuthology.orchestra.run.smithi192.stdout: fsid = d61484e8-a8cb-4030-b6c2-cef28a296bff 2024-09-17T23:30:49.934 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.934 INFO:teuthology.orchestra.run.smithi192.stdout:[osd] 2024-09-17T23:30:49.934 INFO:teuthology.orchestra.run.smithi192.stdout: osd journal size = 100 2024-09-17T23:30:49.934 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.934 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub load threshold = 5.0 2024-09-17T23:30:49.934 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub max interval = 600 2024-09-17T23:30:49.935 INFO:teuthology.orchestra.run.smithi192.stdout: osd mclock profile = high_recovery_ops 2024-09-17T23:30:49.935 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.935 INFO:teuthology.orchestra.run.smithi192.stdout: osd recover clone overlap = true 2024-09-17T23:30:49.935 INFO:teuthology.orchestra.run.smithi192.stdout: osd recovery max chunk = 1048576 2024-09-17T23:30:49.935 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.935 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug shutdown = true 2024-09-17T23:30:49.935 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug op order = true 2024-09-17T23:30:49.935 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify stray on activate = true 2024-09-17T23:30:49.935 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug trim objects = true 2024-09-17T23:30:49.935 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.935 INFO:teuthology.orchestra.run.smithi192.stdout: osd open classes on start = true 2024-09-17T23:30:49.935 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug pg log writeout = true 2024-09-17T23:30:49.936 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.936 INFO:teuthology.orchestra.run.smithi192.stdout: osd deep scrub update digest min age = 30 2024-09-17T23:30:49.936 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.936 INFO:teuthology.orchestra.run.smithi192.stdout: osd map max advance = 10 2024-09-17T23:30:49.936 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.936 INFO:teuthology.orchestra.run.smithi192.stdout: journal zero on create = true 2024-09-17T23:30:49.936 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.936 INFO:teuthology.orchestra.run.smithi192.stdout: filestore ondisk finisher threads = 3 2024-09-17T23:30:49.936 INFO:teuthology.orchestra.run.smithi192.stdout: filestore apply finisher threads = 3 2024-09-17T23:30:49.936 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.936 INFO:teuthology.orchestra.run.smithi192.stdout: bdev debug aio = true 2024-09-17T23:30:49.936 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug misdirected ops = true 2024-09-17T23:30:49.937 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore block size = 96636764160 2024-09-17T23:30:49.937 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore compression algorithm = zstd 2024-09-17T23:30:49.937 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore compression mode = aggressive 2024-09-17T23:30:49.937 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore fsck on mount = True 2024-09-17T23:30:49.937 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluefs = 1/20 2024-09-17T23:30:49.937 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluestore = 1/20 2024-09-17T23:30:49.937 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-09-17T23:30:49.937 INFO:teuthology.orchestra.run.smithi192.stdout: debug osd = 20 2024-09-17T23:30:49.937 INFO:teuthology.orchestra.run.smithi192.stdout: debug rocksdb = 4/10 2024-09-17T23:30:49.937 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd backfillfull_ratio = 0.85 2024-09-17T23:30:49.937 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd full ratio = 0.9 2024-09-17T23:30:49.937 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd nearfull ratio = 0.8 2024-09-17T23:30:49.937 INFO:teuthology.orchestra.run.smithi192.stdout: osd failsafe full ratio = 0.95 2024-09-17T23:30:49.938 INFO:teuthology.orchestra.run.smithi192.stdout: osd objectstore = bluestore 2024-09-17T23:30:49.938 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.938 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr] 2024-09-17T23:30:49.938 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-09-17T23:30:49.938 INFO:teuthology.orchestra.run.smithi192.stdout: debug mgr = 20 2024-09-17T23:30:49.938 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-09-17T23:30:49.938 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-09-17T23:30:49.938 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:30:49.938 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:30:49.938 INFO:teuthology.orchestra.run.smithi192.stdout: mgr/telemetry/nag = false 2024-09-17T23:30:49.938 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.938 INFO:teuthology.orchestra.run.smithi192.stdout:[mon] 2024-09-17T23:30:49.938 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-09-17T23:30:49.938 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-09-17T23:30:49.939 INFO:teuthology.orchestra.run.smithi192.stdout: debug paxos = 20 2024-09-17T23:30:49.939 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-09-17T23:30:49.939 INFO:teuthology.orchestra.run.smithi192.stdout: mon data avail warn = 5 2024-09-17T23:30:49.939 INFO:teuthology.orchestra.run.smithi192.stdout: mon mgr mkfs grace = 240 2024-09-17T23:30:49.939 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-09-17T23:30:49.939 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd reporter subtree level = osd 2024-09-17T23:30:49.939 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd prime pg temp = true 2024-09-17T23:30:49.939 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-09-17T23:30:49.939 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.939 INFO:teuthology.orchestra.run.smithi192.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-17T23:30:49.939 INFO:teuthology.orchestra.run.smithi192.stdout: auth mon ticket ttl = 660 # 11m 2024-09-17T23:30:49.939 INFO:teuthology.orchestra.run.smithi192.stdout: auth service ticket ttl = 240 # 4m 2024-09-17T23:30:49.939 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.940 INFO:teuthology.orchestra.run.smithi192.stdout: # don't complain about insecure global_id in the test suite 2024-09-17T23:30:49.940 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-17T23:30:49.940 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-17T23:30:49.940 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.940 INFO:teuthology.orchestra.run.smithi192.stdout: # 1m isn't quite enough 2024-09-17T23:30:49.940 INFO:teuthology.orchestra.run.smithi192.stdout: mon_down_mkfs_grace = 2m 2024-09-17T23:30:49.940 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.940 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_filestore_osds = false 2024-09-17T23:30:49.940 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:30:49.940 INFO:teuthology.orchestra.run.smithi192.stdout:[client] 2024-09-17T23:30:49.940 INFO:teuthology.orchestra.run.smithi192.stdout: rgw cache enabled = true 2024-09-17T23:30:49.940 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable ops log = true 2024-09-17T23:30:49.941 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable usage log = true 2024-09-17T23:30:49.941 INFO:teuthology.orchestra.run.smithi192.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-17T23:30:49.941 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-17T23:30:49.941 INFO:teuthology.orchestra.run.smithi192.stdout: debug rbd = 20 2024-09-17T23:30:49.941 INFO:teuthology.orchestra.run.smithi192.stdout: debug rbd_mirror = 15 2024-09-17T23:30:49.941 INFO:teuthology.orchestra.run.smithi192.stdout: log to stderr = False 2024-09-17T23:30:49.941 INFO:teuthology.orchestra.run.smithi192.stdout: rbd default features = 125 2024-09-17T23:30:49.941 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.a] 2024-09-17T23:30:49.941 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.0] 2024-09-17T23:30:49.941 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.941 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.941 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.1] 2024-09-17T23:30:49.941 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.942 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.942 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.2] 2024-09-17T23:30:49.942 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.942 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.942 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.3] 2024-09-17T23:30:49.942 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.942 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.942 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.4] 2024-09-17T23:30:49.942 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.942 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.942 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.5] 2024-09-17T23:30:49.942 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.942 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.943 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.6] 2024-09-17T23:30:49.943 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-17T23:30:49.943 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-17T23:30:49.944 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-17T23:30:49.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-17T23:30:50.039 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-17T23:30:50.039 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:50.039 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-17T23:30:50.054 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:50.055 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-17T23:30:50.111 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi096.front.sepia.ceph.com 2024-09-17T23:30:50.111 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:50.111 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-17T23:30:50.111 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-17T23:30:50.153 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:30:50.153 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:30:50.211 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi192.front.sepia.ceph.com 2024-09-17T23:30:50.211 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:50.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-17T23:30:50.211 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-17T23:30:50.246 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:30:50.246 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:30:50.301 INFO:tasks.ceph:Setting up mon nodes... 2024-09-17T23:30:50.302 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-17T23:30:50.302 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-17T23:30:50.359 INFO:teuthology.orchestra.run.smithi096.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-17T23:30:50.362 INFO:tasks.ceph:Setting up mds nodes... 2024-09-17T23:30:50.362 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-17T23:30:50.362 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-17T23:30:50.406 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-17T23:30:50.419 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-17T23:30:50.460 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-17T23:30:50.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-17T23:30:50.557 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-17T23:30:50.570 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-17T23:30:50.651 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-17T23:30:50.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-17T23:30:50.745 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-17T23:30:50.757 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-17T23:30:50.839 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-17T23:30:50.851 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-17T23:30:50.931 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-17T23:30:50.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-17T23:30:51.025 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-17T23:30:51.038 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-17T23:30:51.120 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-17T23:30:51.132 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-17T23:30:51.132 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-17T23:30:51.132 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-17T23:30:51.197 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:30:51.197 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-17T23:30:51.197 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com 2024-09-17T23:30:51.197 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-17T23:30:51.269 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T23:30:51.269 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T23:30:51.269 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T23:30:51.269 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T23:30:51.269 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T23:30:51.269 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-09-17T23:30:51.269 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T23:30:51.269 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T23:30:51.269 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T23:30:51.269 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T23:30:51.781 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-09-17T23:30:51.783 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-09-17T23:30:51.783 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-17T23:30:51.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-17T23:30:51.964 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-17T23:30:52.030 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:30:52.030 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-17T23:30:52.031 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com 2024-09-17T23:30:52.031 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-17T23:30:52.102 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T23:30:52.102 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T23:30:52.103 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T23:30:52.103 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T23:30:52.103 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T23:30:52.103 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-09-17T23:30:52.103 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T23:30:52.103 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T23:30:52.103 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T23:30:52.103 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T23:30:52.542 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-09-17T23:30:52.544 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-09-17T23:30:52.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-17T23:30:52.629 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-17T23:30:52.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-17T23:30:52.765 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-17T23:30:52.765 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-17T23:30:52.765 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com 2024-09-17T23:30:52.765 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-17T23:30:52.838 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-17T23:30:52.838 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-17T23:30:52.838 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-17T23:30:52.838 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-17T23:30:52.838 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-17T23:30:52.838 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-09-17T23:30:52.838 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-17T23:30:52.838 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-17T23:30:52.838 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-17T23:30:52.838 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-17T23:30:53.365 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-09-17T23:30:53.367 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-09-17T23:30:53.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-17T23:30:53.509 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-17T23:30:53.540 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:30:53.667 INFO:teuthology.orchestra.run.smithi192.stderr:2024-09-17T23:30:53.665+0000 7f99dd00b640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-17T23:30:53.669 INFO:teuthology.orchestra.run.smithi192.stderr:2024-09-17T23:30:53.667+0000 7f99dd00b640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-17T23:30:53.669 INFO:teuthology.orchestra.run.smithi192.stderr:2024-09-17T23:30:53.667+0000 7f99dd00b640 -1 bdev(0x556b914ef800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2024-09-17T23:30:53.670 INFO:teuthology.orchestra.run.smithi192.stderr:2024-09-17T23:30:53.667+0000 7f99dd00b640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-17T23:30:56.582 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-17T23:30:56.613 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:30:56.703 INFO:teuthology.orchestra.run.smithi192.stderr:2024-09-17T23:30:56.701+0000 7f312ba0d640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-17T23:30:56.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-09-17T23:30:56.701+0000 7f312ba0d640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-17T23:30:56.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-09-17T23:30:56.701+0000 7f312ba0d640 -1 bdev(0x55fd3967f800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2024-09-17T23:30:56.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-09-17T23:30:56.702+0000 7f312ba0d640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-17T23:30:59.608 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-17T23:30:59.638 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:30:59.729 INFO:teuthology.orchestra.run.smithi192.stderr:2024-09-17T23:30:59.726+0000 7f576e62f640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-17T23:30:59.729 INFO:teuthology.orchestra.run.smithi192.stderr:2024-09-17T23:30:59.726+0000 7f576e62f640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-17T23:30:59.729 INFO:teuthology.orchestra.run.smithi192.stderr:2024-09-17T23:30:59.726+0000 7f576e62f640 -1 bdev(0x55ebfd1af800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2024-09-17T23:30:59.729 INFO:teuthology.orchestra.run.smithi192.stderr:2024-09-17T23:30:59.727+0000 7f576e62f640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-17T23:31:02.645 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-17T23:31:02.675 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-17T23:31:02.676 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:31:02.676 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-17T23:31:02.707 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:31:02.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-17T23:31:02.744 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:31:02.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-17T23:31:02.813 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:31:02.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-17T23:31:02.880 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-17T23:31:02.880 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-17T23:31:02.899 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:31:02.899 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-17T23:31:02.936 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:31:02.936 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-17T23:31:02.993 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:31:02.993 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-17T23:31:03.049 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:31:03.049 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-17T23:31:03.105 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:31:03.105 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-17T23:31:03.161 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:31:03.161 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-17T23:31:03.217 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:31:03.217 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-17T23:31:03.274 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:31:03.274 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-17T23:31:03.330 INFO:tasks.ceph:Adding keys to all mons... 2024-09-17T23:31:03.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-17T23:31:03.393 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.x] 2024-09-17T23:31:03.393 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAqEepmrsNhFRAAsCzL8VjfvO8STYe8MKCp8g== 2024-09-17T23:31:03.393 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.0] 2024-09-17T23:31:03.393 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAtEepmt/25JxAAjRg8/dGcze9jztF6XXlPPA== 2024-09-17T23:31:03.393 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.1] 2024-09-17T23:31:03.393 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAwEepmx9XeKRAA300xQkzdZdFHmz214AhABA== 2024-09-17T23:31:03.393 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.2] 2024-09-17T23:31:03.394 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAzEepmxv1dKxAAkWwiqFLRwGx+K5xbvoxhRg== 2024-09-17T23:31:03.394 INFO:teuthology.orchestra.run.smithi192.stdout:[client.0] 2024-09-17T23:31:03.394 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAqEepm6D8oGBAAoG3wLbSeU+BChppZ0+nAVg== 2024-09-17T23:31:03.394 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror] 2024-09-17T23:31:03.394 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAqEepmKxtlGxAAr7+r9sND+XuhB5w22FR3BQ== 2024-09-17T23:31:03.394 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.0] 2024-09-17T23:31:03.394 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAqEepmpa0kIRAAk2F66809M64I90UiIlgm+g== 2024-09-17T23:31:03.394 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.1] 2024-09-17T23:31:03.394 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAqEepmc2C8JhAAGi2VcWL798rh/Z5NbMjBFQ== 2024-09-17T23:31:03.394 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.2] 2024-09-17T23:31:03.394 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAqEepmM+5WLBAA0s0FIZA6HmenUlo0RuLW+A== 2024-09-17T23:31:03.394 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.3] 2024-09-17T23:31:03.394 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAqEepmjCvwMRAAGI3QbV2uYET9CBgfpzm4fw== 2024-09-17T23:31:03.395 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.4] 2024-09-17T23:31:03.395 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAqEepmUId6NxAAcgsX9vA3QJLhMQEhQmJJUw== 2024-09-17T23:31:03.395 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.5] 2024-09-17T23:31:03.395 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQArEepmJEB4ARAAtwge7FrxKXfjiyFJ+vYcqg== 2024-09-17T23:31:03.395 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.6] 2024-09-17T23:31:03.395 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQArEepmCnYVBxAA3CtnLwukM7tYk+M0J53M9A== 2024-09-17T23:31:03.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-17T23:31:03.486 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T23:31:03.538 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T23:31:03.588 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-17T23:31:03.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:31:03.694 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:31:03.748 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:31:03.803 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:31:03.856 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:31:03.910 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:31:03.965 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:31:04.018 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:31:04.072 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-17T23:31:04.125 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-17T23:31:04.125 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-17T23:31:04.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-17T23:31:04.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-17T23:31:04.270 DEBUG:teuthology.orchestra.run.smithi192:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-17T23:31:04.326 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-17T23:31:04.326 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-17T23:31:04.326 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-17T23:31:04.369 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-17T23:31:04.369 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-17T23:31:04.369 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-17T23:31:04.369 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-17T23:31:04.372 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-17T23:31:04.372 INFO:tasks.ceph:Setting crush tunables to default 2024-09-17T23:31:04.372 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-17T23:31:04.540 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:04.538+0000 7fdcfe188040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T23:31:04.541 INFO:teuthology.orchestra.run.smithi192.stderr:adjusted tunables profile to default 2024-09-17T23:31:04.551 INFO:tasks.ceph:check_enable_crimson: False 2024-09-17T23:31:04.552 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-17T23:31:04.552 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:31:04.552 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-17T23:31:04.576 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:31:04.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-17T23:31:04.581 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:04.579+0000 7fdcfe188040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T23:31:04.641 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:31:04.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-17T23:31:04.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster cluster2 osd new dc15472a-cb32-4dcf-a0df-76086c7fed31 0 2024-09-17T23:31:04.711 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:04.709+0000 7fdcfe188040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T23:31:04.758 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:04.756+0000 7fdcfe188040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T23:31:04.864 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:04.862+0000 7fdcfe188040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T23:31:04.922 INFO:teuthology.orchestra.run.smithi192.stdout:0 2024-09-17T23:31:04.933 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster cluster2 osd new 4c9ba9c3-191f-4414-aa4b-46c71c7e21f7 1 2024-09-17T23:31:04.956 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:04.954+0000 7fdcfe188040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T23:31:05.106 INFO:teuthology.orchestra.run.smithi192.stdout:1 2024-09-17T23:31:05.117 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster cluster2 osd new 7d465e64-5b7a-4189-bcce-b13ccb458643 2 2024-09-17T23:31:05.216 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:05.214+0000 7fdcfe188040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T23:31:05.260 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:05.258+0000 7fdcfe188040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T23:31:05.289 INFO:teuthology.orchestra.run.smithi192.stdout:2 2024-09-17T23:31:05.301 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-17T23:31:05.301 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-17T23:31:05.303 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-17T23:31:05.303 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-17T23:31:05.304 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-17T23:31:05.305 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-17T23:31:05.306 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-17T23:31:05.306 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-17T23:31:05.308 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-17T23:31:05.309 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-17T23:31:05.376 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:05.374+0000 7fdcfe188040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T23:31:05.477 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:31:05.477 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":5,"fsid":"d61484e8-a8cb-4030-b6c2-cef28a296bff","created":"2024-09-17T23:31:04.453616+0000","modified":"2024-09-17T23:31:05.288256+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T23:31:05.288256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dc15472a-cb32-4dcf-a0df-76086c7fed31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4c9ba9c3-191f-4414-aa4b-46c71c7e21f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d465e64-5b7a-4189-bcce-b13ccb458643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:31:05.488 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-17T23:31:05.489 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-17T23:31:05.792 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-17T23:31:05.949 INFO:teuthology.misc.health.smithi192.stdout: 2024-09-17T23:31:05.949 INFO:teuthology.misc.health.smithi192.stdout:{"epoch":5,"fsid":"d61484e8-a8cb-4030-b6c2-cef28a296bff","created":"2024-09-17T23:31:04.453616+0000","modified":"2024-09-17T23:31:05.288256+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T23:31:05.288256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dc15472a-cb32-4dcf-a0df-76086c7fed31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4c9ba9c3-191f-4414-aa4b-46c71c7e21f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d465e64-5b7a-4189-bcce-b13ccb458643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:31:05.958 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-17T23:31:06.069 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:06.067+0000 7fdcfe188040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T23:31:06.112 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:06.110+0000 7fdcfe188040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T23:31:06.418 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:06.416+0000 7fdcfe188040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T23:31:06.500 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:06.498+0000 7fdcfe188040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T23:31:06.504 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-17T23:31:06.502+0000 7fd423e15640 -1 Falling back to public interface 2024-09-17T23:31:06.607 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:06.604+0000 7fdcfe188040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T23:31:06.649 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:06.647+0000 7fdcfe188040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T23:31:06.741 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:31:06.738+0000 7fc286264640 -1 Falling back to public interface 2024-09-17T23:31:06.758 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:06.756+0000 7fdcfe188040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T23:31:06.764 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-17T23:31:06.762+0000 7f0124945640 -1 Falling back to public interface 2024-09-17T23:31:06.809 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:06.807+0000 7fdcfe188040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T23:31:06.850 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:06.847+0000 7fdcfe188040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T23:31:06.962 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:06.960+0000 7fdcfe188040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T23:31:07.010 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:07.008+0000 7fdcfe188040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T23:31:07.058 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:07.056+0000 7fdcfe188040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T23:31:07.110 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:07.108+0000 7fdcfe188040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T23:31:07.152 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:07.149+0000 7fdcfe188040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T23:31:07.193 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:07.191+0000 7fdcfe188040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T23:31:07.751 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:07.748+0000 7fdcfe188040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T23:31:08.759 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-17T23:31:08.757+0000 7fdc9ea86640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-17T23:31:10.546 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-17T23:31:10.544+0000 7fd423e15640 -1 osd.1 0 log_to_monitors true 2024-09-17T23:31:10.750 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:31:10.748+0000 7fc286264640 -1 osd.0 0 log_to_monitors true 2024-09-17T23:31:11.288 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-17T23:31:11.286+0000 7f0124945640 -1 osd.2 0 log_to_monitors true 2024-09-17T23:31:12.261 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-17T23:31:12.642 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-17T23:31:12.640+0000 7fd41ff1a640 -1 osd.1 0 waiting for initial osdmap 2024-09-17T23:31:12.648 INFO:teuthology.misc.health.smithi192.stdout: 2024-09-17T23:31:12.648 INFO:teuthology.misc.health.smithi192.stdout:{"epoch":7,"fsid":"d61484e8-a8cb-4030-b6c2-cef28a296bff","created":"2024-09-17T23:31:04.453616+0000","modified":"2024-09-17T23:31:11.757785+0000","last_up_change":"0.000000","last_in_change":"2024-09-17T23:31:05.288256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dc15472a-cb32-4dcf-a0df-76086c7fed31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4c9ba9c3-191f-4414-aa4b-46c71c7e21f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d465e64-5b7a-4189-bcce-b13ccb458643","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:31:12.660 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-17T23:31:12.660 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-17T23:31:12.658+0000 7fd417526640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T23:31:12.678 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:31:12.676+0000 7fc2821f3640 -1 osd.0 0 waiting for initial osdmap 2024-09-17T23:31:12.694 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:31:12.692+0000 7fc27a013640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T23:31:13.467 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-17T23:31:13.465+0000 7f01208d4640 -1 osd.2 0 waiting for initial osdmap 2024-09-17T23:31:13.477 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-17T23:31:13.475+0000 7f01186f4640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T23:31:18.963 DEBUG:teuthology.orchestra.run.smithi192:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-17T23:31:19.245 INFO:teuthology.misc.health.smithi192.stdout: 2024-09-17T23:31:19.245 INFO:teuthology.misc.health.smithi192.stdout:{"epoch":11,"fsid":"d61484e8-a8cb-4030-b6c2-cef28a296bff","created":"2024-09-17T23:31:04.453616+0000","modified":"2024-09-17T23:31:16.469173+0000","last_up_change":"2024-09-17T23:31:14.466759+0000","last_in_change":"2024-09-17T23:31:05.288256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:31:13.785771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dc15472a-cb32-4dcf-a0df-76086c7fed31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6809","nonce":1545791238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6811","nonce":1545791238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6815","nonce":1545791238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6813","nonce":1545791238}]},"public_addr":"172.21.15.192:6809/1545791238","cluster_addr":"172.21.15.192:6811/1545791238","heartbeat_back_addr":"172.21.15.192:6815/1545791238","heartbeat_front_addr":"172.21.15.192:6813/1545791238","state":["exists","up"]},{"osd":1,"uuid":"4c9ba9c3-191f-4414-aa4b-46c71c7e21f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6801","nonce":3313808101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6803","nonce":3313808101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6807","nonce":3313808101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6805","nonce":3313808101}]},"public_addr":"172.21.15.192:6801/3313808101","cluster_addr":"172.21.15.192:6803/3313808101","heartbeat_back_addr":"172.21.15.192:6807/3313808101","heartbeat_front_addr":"172.21.15.192:6805/3313808101","state":["exists","up"]},{"osd":2,"uuid":"7d465e64-5b7a-4189-bcce-b13ccb458643","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6817","nonce":1728261310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6819","nonce":1728261310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6823","nonce":1728261310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6821","nonce":1728261310}]},"public_addr":"172.21.15.192:6817/1728261310","cluster_addr":"172.21.15.192:6819/1728261310","heartbeat_back_addr":"172.21.15.192:6823/1728261310","heartbeat_front_addr":"172.21.15.192:6821/1728261310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:31:11.739048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:31:11.591739+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:31:12.300977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:31:19.257 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-17T23:31:19.257 INFO:tasks.ceph:Creating RBD pool 2024-09-17T23:31:19.257 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-17T23:31:20.482 INFO:teuthology.orchestra.run.smithi192.stderr:pool 'rbd' created 2024-09-17T23:31:20.500 DEBUG:teuthology.orchestra.run.smithi192:> rbd --cluster cluster2 pool init rbd 2024-09-17T23:31:23.519 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-17T23:31:23.520 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-17T23:31:23.520 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-17T23:31:23.520 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-17T23:31:23.520 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-17T23:31:23.851 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:31:23.862 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":295482559},{"type":"v1","addr":"172.21.15.96:6827","nonce":295482559}]},"active_addr":"172.21.15.96:6827/295482559","active_change":"2024-09-17T23:31:07.751357+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":4021323237}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":1017526642}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":3552180706}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":696703905}]}]} 2024-09-17T23:31:23.864 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-17T23:31:23.865 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-17T23:31:23.865 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-17T23:31:24.140 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:31:24.140 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":15,"fsid":"d61484e8-a8cb-4030-b6c2-cef28a296bff","created":"2024-09-17T23:31:04.453616+0000","modified":"2024-09-17T23:31:23.497014+0000","last_up_change":"2024-09-17T23:31:14.466759+0000","last_in_change":"2024-09-17T23:31:05.288256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:31:13.785771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T23:31:19.553055+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dc15472a-cb32-4dcf-a0df-76086c7fed31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6809","nonce":1545791238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6811","nonce":1545791238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6815","nonce":1545791238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6813","nonce":1545791238}]},"public_addr":"172.21.15.192:6809/1545791238","cluster_addr":"172.21.15.192:6811/1545791238","heartbeat_back_addr":"172.21.15.192:6815/1545791238","heartbeat_front_addr":"172.21.15.192:6813/1545791238","state":["exists","up"]},{"osd":1,"uuid":"4c9ba9c3-191f-4414-aa4b-46c71c7e21f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6801","nonce":3313808101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6803","nonce":3313808101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6807","nonce":3313808101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6805","nonce":3313808101}]},"public_addr":"172.21.15.192:6801/3313808101","cluster_addr":"172.21.15.192:6803/3313808101","heartbeat_back_addr":"172.21.15.192:6807/3313808101","heartbeat_front_addr":"172.21.15.192:6805/3313808101","state":["exists","up"]},{"osd":2,"uuid":"7d465e64-5b7a-4189-bcce-b13ccb458643","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6817","nonce":1728261310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6819","nonce":1728261310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6823","nonce":1728261310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6821","nonce":1728261310}]},"public_addr":"172.21.15.192:6817/1728261310","cluster_addr":"172.21.15.192:6819/1728261310","heartbeat_back_addr":"172.21.15.192:6823/1728261310","heartbeat_front_addr":"172.21.15.192:6821/1728261310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:31:11.739048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:31:11.591739+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:31:12.300977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:31:24.152 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-17T23:31:24.152 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-17T23:31:24.429 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:31:24.429 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":15,"fsid":"d61484e8-a8cb-4030-b6c2-cef28a296bff","created":"2024-09-17T23:31:04.453616+0000","modified":"2024-09-17T23:31:23.497014+0000","last_up_change":"2024-09-17T23:31:14.466759+0000","last_in_change":"2024-09-17T23:31:05.288256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:31:13.785771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T23:31:19.553055+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dc15472a-cb32-4dcf-a0df-76086c7fed31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6809","nonce":1545791238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6811","nonce":1545791238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6815","nonce":1545791238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6813","nonce":1545791238}]},"public_addr":"172.21.15.192:6809/1545791238","cluster_addr":"172.21.15.192:6811/1545791238","heartbeat_back_addr":"172.21.15.192:6815/1545791238","heartbeat_front_addr":"172.21.15.192:6813/1545791238","state":["exists","up"]},{"osd":1,"uuid":"4c9ba9c3-191f-4414-aa4b-46c71c7e21f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6801","nonce":3313808101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6803","nonce":3313808101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6807","nonce":3313808101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6805","nonce":3313808101}]},"public_addr":"172.21.15.192:6801/3313808101","cluster_addr":"172.21.15.192:6803/3313808101","heartbeat_back_addr":"172.21.15.192:6807/3313808101","heartbeat_front_addr":"172.21.15.192:6805/3313808101","state":["exists","up"]},{"osd":2,"uuid":"7d465e64-5b7a-4189-bcce-b13ccb458643","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6817","nonce":1728261310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6819","nonce":1728261310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6823","nonce":1728261310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6821","nonce":1728261310}]},"public_addr":"172.21.15.192:6817/1728261310","cluster_addr":"172.21.15.192:6819/1728261310","heartbeat_back_addr":"172.21.15.192:6823/1728261310","heartbeat_front_addr":"172.21.15.192:6821/1728261310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:31:11.739048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:31:11.591739+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:31:12.300977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T23:31:24.442 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-17T23:31:24.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-17T23:31:24.443 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-17T23:31:24.580 INFO:teuthology.orchestra.run.smithi192.stdout:34359738373 2024-09-17T23:31:24.580 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-17T23:31:24.598 INFO:teuthology.orchestra.run.smithi192.stdout:34359738374 2024-09-17T23:31:24.599 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-17T23:31:24.600 INFO:teuthology.orchestra.run.smithi192.stdout:38654705669 2024-09-17T23:31:24.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-17T23:31:24.894 INFO:teuthology.orchestra.run.smithi192.stdout:34359738372 2024-09-17T23:31:24.908 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2024-09-17T23:31:24.932 INFO:teuthology.orchestra.run.smithi192.stdout:34359738372 2024-09-17T23:31:24.945 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.1 2024-09-17T23:31:24.960 INFO:teuthology.orchestra.run.smithi192.stdout:38654705668 2024-09-17T23:31:24.973 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705668 for osd.2 2024-09-17T23:31:25.909 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-17T23:31:25.946 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-17T23:31:25.974 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-17T23:31:26.228 INFO:teuthology.orchestra.run.smithi192.stdout:34359738372 2024-09-17T23:31:26.241 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2024-09-17T23:31:26.268 INFO:teuthology.orchestra.run.smithi192.stdout:34359738372 2024-09-17T23:31:26.281 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738372 for osd.1 2024-09-17T23:31:26.291 INFO:teuthology.orchestra.run.smithi192.stdout:38654705668 2024-09-17T23:31:26.303 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705668 for osd.2 2024-09-17T23:31:27.243 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-17T23:31:27.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-17T23:31:27.305 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-17T23:31:27.579 INFO:teuthology.orchestra.run.smithi192.stdout:34359738374 2024-09-17T23:31:27.590 INFO:teuthology.orchestra.run.smithi192.stdout:34359738373 2024-09-17T23:31:27.592 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2024-09-17T23:31:27.593 DEBUG:teuthology.parallel:result is None 2024-09-17T23:31:27.603 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.0 2024-09-17T23:31:27.603 DEBUG:teuthology.parallel:result is None 2024-09-17T23:31:27.655 INFO:teuthology.orchestra.run.smithi192.stdout:38654705670 2024-09-17T23:31:27.668 INFO:tasks.ceph.ceph_manager.cluster2:need seq 38654705669 got 38654705670 for osd.2 2024-09-17T23:31:27.668 DEBUG:teuthology.parallel:result is None 2024-09-17T23:31:27.668 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-17T23:31:27.668 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-17T23:31:27.943 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:31:27.943 INFO:teuthology.orchestra.run.smithi192.stderr:dumped all 2024-09-17T23:31:27.955 INFO:teuthology.orchestra.run.smithi192.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-17T23:31:27.758059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":110,"num_read_kb":95,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81596,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033924,"statfs":{"total":289910292480,"available":289826738176,"internally_reserved":0,"allocated":516096,"data_stored":1296850,"data_compressed":5800,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.272767"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:23.507145+0000","last_change":"2024-09-17T23:31:23.507145+0000","last_active":"2024-09-17T23:31:23.507145+0000","last_peered":"2024-09-17T23:31:23.507145+0000","last_clean":"2024-09-17T23:31:23.507145+0000","last_became_active":"2024-09-17T23:31:21.498864+0000","last_became_peered":"2024-09-17T23:31:21.498864+0000","last_unstale":"2024-09-17T23:31:23.507145+0000","last_undegraded":"2024-09-17T23:31:23.507145+0000","last_fullsized":"2024-09-17T23:31:23.507145+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:08:07.786924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083165799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:23.507426+0000","last_change":"2024-09-17T23:31:23.507426+0000","last_active":"2024-09-17T23:31:23.507426+0000","last_peered":"2024-09-17T23:31:23.507426+0000","last_clean":"2024-09-17T23:31:23.507426+0000","last_became_active":"2024-09-17T23:31:21.498384+0000","last_became_peered":"2024-09-17T23:31:21.498384+0000","last_unstale":"2024-09-17T23:31:23.507426+0000","last_undegraded":"2024-09-17T23:31:23.507426+0000","last_fullsized":"2024-09-17T23:31:23.507426+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:36:11.256952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010814310000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:23.506777+0000","last_change":"2024-09-17T23:31:23.506777+0000","last_active":"2024-09-17T23:31:23.506777+0000","last_peered":"2024-09-17T23:31:23.506777+0000","last_clean":"2024-09-17T23:31:23.506777+0000","last_became_active":"2024-09-17T23:31:21.497851+0000","last_became_peered":"2024-09-17T23:31:21.497851+0000","last_unstale":"2024-09-17T23:31:23.506777+0000","last_undegraded":"2024-09-17T23:31:23.506777+0000","last_fullsized":"2024-09-17T23:31:23.506777+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:30:54.399154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049346500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:23.507092+0000","last_change":"2024-09-17T23:31:23.507092+0000","last_active":"2024-09-17T23:31:23.507092+0000","last_peered":"2024-09-17T23:31:23.507092+0000","last_clean":"2024-09-17T23:31:23.507092+0000","last_became_active":"2024-09-17T23:31:21.497623+0000","last_became_peered":"2024-09-17T23:31:21.497623+0000","last_unstale":"2024-09-17T23:31:23.507092+0000","last_undegraded":"2024-09-17T23:31:23.507092+0000","last_fullsized":"2024-09-17T23:31:23.507092+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:32:29.760120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079303000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:23.511202+0000","last_change":"2024-09-17T23:31:23.511202+0000","last_active":"2024-09-17T23:31:23.511202+0000","last_peered":"2024-09-17T23:31:23.511202+0000","last_clean":"2024-09-17T23:31:23.511202+0000","last_became_active":"2024-09-17T23:31:21.498502+0000","last_became_peered":"2024-09-17T23:31:21.498502+0000","last_unstale":"2024-09-17T23:31:23.511202+0000","last_undegraded":"2024-09-17T23:31:23.511202+0000","last_fullsized":"2024-09-17T23:31:23.511202+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:08:28.112030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094839100000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:24.091376+0000","last_change":"2024-09-17T23:31:24.091376+0000","last_active":"2024-09-17T23:31:24.091376+0000","last_peered":"2024-09-17T23:31:24.091376+0000","last_clean":"2024-09-17T23:31:24.091376+0000","last_became_active":"2024-09-17T23:31:21.496953+0000","last_became_peered":"2024-09-17T23:31:21.496953+0000","last_unstale":"2024-09-17T23:31:24.091376+0000","last_undegraded":"2024-09-17T23:31:24.091376+0000","last_fullsized":"2024-09-17T23:31:24.091376+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:40:43.130208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031250800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:24.091323+0000","last_change":"2024-09-17T23:31:24.091323+0000","last_active":"2024-09-17T23:31:24.091323+0000","last_peered":"2024-09-17T23:31:24.091323+0000","last_clean":"2024-09-17T23:31:24.091323+0000","last_became_active":"2024-09-17T23:31:21.497936+0000","last_became_peered":"2024-09-17T23:31:21.497936+0000","last_unstale":"2024-09-17T23:31:24.091323+0000","last_undegraded":"2024-09-17T23:31:24.091323+0000","last_fullsized":"2024-09-17T23:31:24.091323+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:22:43.988917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030225,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:23.506776+0000","last_change":"2024-09-17T23:31:23.506776+0000","last_active":"2024-09-17T23:31:23.506776+0000","last_peered":"2024-09-17T23:31:23.506776+0000","last_clean":"2024-09-17T23:31:23.506776+0000","last_became_active":"2024-09-17T23:31:21.497054+0000","last_became_peered":"2024-09-17T23:31:21.497054+0000","last_unstale":"2024-09-17T23:31:23.506776+0000","last_undegraded":"2024-09-17T23:31:23.506776+0000","last_fullsized":"2024-09-17T23:31:23.506776+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:43:27.049283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051159800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'97","reported_seq":148,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:23.506133+0000","last_change":"2024-09-17T23:31:15.637809+0000","last_active":"2024-09-17T23:31:23.506133+0000","last_peered":"2024-09-17T23:31:23.506133+0000","last_clean":"2024-09-17T23:31:23.506133+0000","last_became_active":"2024-09-17T23:31:15.636714+0000","last_became_peered":"2024-09-17T23:31:15.636714+0000","last_unstale":"2024-09-17T23:31:23.506133+0000","last_undegraded":"2024-09-17T23:31:23.506133+0000","last_fullsized":"2024-09-17T23:31:23.506133+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:14.466759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:14.466759+0000","last_clean_scrub_stamp":"2024-09-17T23:31:14.466759+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:05:04.207207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":95,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":95,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5800,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":38692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":2900,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":2900,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2900,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2900,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:31:27.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-17T23:31:28.228 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:31:28.229 INFO:teuthology.orchestra.run.smithi192.stderr:dumped all 2024-09-17T23:31:28.240 INFO:teuthology.orchestra.run.smithi192.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-17T23:31:27.758059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":110,"num_read_kb":95,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81596,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033924,"statfs":{"total":289910292480,"available":289826738176,"internally_reserved":0,"allocated":516096,"data_stored":1296850,"data_compressed":5800,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.272767"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:23.507145+0000","last_change":"2024-09-17T23:31:23.507145+0000","last_active":"2024-09-17T23:31:23.507145+0000","last_peered":"2024-09-17T23:31:23.507145+0000","last_clean":"2024-09-17T23:31:23.507145+0000","last_became_active":"2024-09-17T23:31:21.498864+0000","last_became_peered":"2024-09-17T23:31:21.498864+0000","last_unstale":"2024-09-17T23:31:23.507145+0000","last_undegraded":"2024-09-17T23:31:23.507145+0000","last_fullsized":"2024-09-17T23:31:23.507145+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:08:07.786924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083165799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:23.507426+0000","last_change":"2024-09-17T23:31:23.507426+0000","last_active":"2024-09-17T23:31:23.507426+0000","last_peered":"2024-09-17T23:31:23.507426+0000","last_clean":"2024-09-17T23:31:23.507426+0000","last_became_active":"2024-09-17T23:31:21.498384+0000","last_became_peered":"2024-09-17T23:31:21.498384+0000","last_unstale":"2024-09-17T23:31:23.507426+0000","last_undegraded":"2024-09-17T23:31:23.507426+0000","last_fullsized":"2024-09-17T23:31:23.507426+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:36:11.256952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010814310000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:23.506777+0000","last_change":"2024-09-17T23:31:23.506777+0000","last_active":"2024-09-17T23:31:23.506777+0000","last_peered":"2024-09-17T23:31:23.506777+0000","last_clean":"2024-09-17T23:31:23.506777+0000","last_became_active":"2024-09-17T23:31:21.497851+0000","last_became_peered":"2024-09-17T23:31:21.497851+0000","last_unstale":"2024-09-17T23:31:23.506777+0000","last_undegraded":"2024-09-17T23:31:23.506777+0000","last_fullsized":"2024-09-17T23:31:23.506777+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:30:54.399154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049346500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:23.507092+0000","last_change":"2024-09-17T23:31:23.507092+0000","last_active":"2024-09-17T23:31:23.507092+0000","last_peered":"2024-09-17T23:31:23.507092+0000","last_clean":"2024-09-17T23:31:23.507092+0000","last_became_active":"2024-09-17T23:31:21.497623+0000","last_became_peered":"2024-09-17T23:31:21.497623+0000","last_unstale":"2024-09-17T23:31:23.507092+0000","last_undegraded":"2024-09-17T23:31:23.507092+0000","last_fullsized":"2024-09-17T23:31:23.507092+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:32:29.760120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079303000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:23.511202+0000","last_change":"2024-09-17T23:31:23.511202+0000","last_active":"2024-09-17T23:31:23.511202+0000","last_peered":"2024-09-17T23:31:23.511202+0000","last_clean":"2024-09-17T23:31:23.511202+0000","last_became_active":"2024-09-17T23:31:21.498502+0000","last_became_peered":"2024-09-17T23:31:21.498502+0000","last_unstale":"2024-09-17T23:31:23.511202+0000","last_undegraded":"2024-09-17T23:31:23.511202+0000","last_fullsized":"2024-09-17T23:31:23.511202+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:08:28.112030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094839100000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:24.091376+0000","last_change":"2024-09-17T23:31:24.091376+0000","last_active":"2024-09-17T23:31:24.091376+0000","last_peered":"2024-09-17T23:31:24.091376+0000","last_clean":"2024-09-17T23:31:24.091376+0000","last_became_active":"2024-09-17T23:31:21.496953+0000","last_became_peered":"2024-09-17T23:31:21.496953+0000","last_unstale":"2024-09-17T23:31:24.091376+0000","last_undegraded":"2024-09-17T23:31:24.091376+0000","last_fullsized":"2024-09-17T23:31:24.091376+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:40:43.130208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031250800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:24.091323+0000","last_change":"2024-09-17T23:31:24.091323+0000","last_active":"2024-09-17T23:31:24.091323+0000","last_peered":"2024-09-17T23:31:24.091323+0000","last_clean":"2024-09-17T23:31:24.091323+0000","last_became_active":"2024-09-17T23:31:21.497936+0000","last_became_peered":"2024-09-17T23:31:21.497936+0000","last_unstale":"2024-09-17T23:31:24.091323+0000","last_undegraded":"2024-09-17T23:31:24.091323+0000","last_fullsized":"2024-09-17T23:31:24.091323+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:22:43.988917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030225,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:23.506776+0000","last_change":"2024-09-17T23:31:23.506776+0000","last_active":"2024-09-17T23:31:23.506776+0000","last_peered":"2024-09-17T23:31:23.506776+0000","last_clean":"2024-09-17T23:31:23.506776+0000","last_became_active":"2024-09-17T23:31:21.497054+0000","last_became_peered":"2024-09-17T23:31:21.497054+0000","last_unstale":"2024-09-17T23:31:23.506776+0000","last_undegraded":"2024-09-17T23:31:23.506776+0000","last_fullsized":"2024-09-17T23:31:23.506776+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:43:27.049283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051159800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'97","reported_seq":148,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-17T23:31:23.506133+0000","last_change":"2024-09-17T23:31:15.637809+0000","last_active":"2024-09-17T23:31:23.506133+0000","last_peered":"2024-09-17T23:31:23.506133+0000","last_clean":"2024-09-17T23:31:23.506133+0000","last_became_active":"2024-09-17T23:31:15.636714+0000","last_became_peered":"2024-09-17T23:31:15.636714+0000","last_unstale":"2024-09-17T23:31:23.506133+0000","last_undegraded":"2024-09-17T23:31:23.506133+0000","last_fullsized":"2024-09-17T23:31:23.506133+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:14.466759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:14.466759+0000","last_clean_scrub_stamp":"2024-09-17T23:31:14.466759+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:05:04.207207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":95,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":95,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":5800,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":38692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":2900,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":2900,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2900,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":2900,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T23:31:28.241 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-17T23:31:28.241 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-17T23:31:28.241 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-17T23:31:28.241 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-17T23:31:28.552 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-17T23:31:28.552 INFO:teuthology.orchestra.run.smithi192.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T23:31:28.565 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-17T23:31:28.565 INFO:teuthology.run_tasks:Running task exec... 2024-09-17T23:31:28.579 INFO:teuthology.task.exec:Executing custom commands... 2024-09-17T23:31:28.579 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi192.front.sepia.ceph.com 2024-09-17T23:31:28.579 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:28.933 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.4] 2024-09-17T23:31:28.933 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQACEepmGL9WDRAAjFpVnfdj7UbJe7JYT9BXAw== 2024-09-17T23:31:28.933 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:31:28.933 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:28.934 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.4 2024-09-17T23:31:28.947 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:29.297 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.5] 2024-09-17T23:31:29.297 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQACEepmlOPnEhAA+3E7oaenLLIcoL0eom0FZg== 2024-09-17T23:31:29.297 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:31:29.298 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:29.298 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.5 2024-09-17T23:31:29.311 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:29.662 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.6] 2024-09-17T23:31:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQACEepmpzh5GBAAph9fanNN6vZwfCedX95rRA== 2024-09-17T23:31:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:31:29.663 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:29.663 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.6 2024-09-17T23:31:29.676 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:30.027 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.4] 2024-09-17T23:31:30.027 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAqEepmUId6NxAAcgsX9vA3QJLhMQEhQmJJUw== 2024-09-17T23:31:30.027 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:31:30.027 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:30.028 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.4 2024-09-17T23:31:30.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:30.387 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.5] 2024-09-17T23:31:30.388 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQArEepmJEB4ARAAtwge7FrxKXfjiyFJ+vYcqg== 2024-09-17T23:31:30.388 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:31:30.388 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:30.388 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.5 2024-09-17T23:31:30.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:30.755 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.6] 2024-09-17T23:31:30.755 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQArEepmCnYVBxAA3CtnLwukM7tYk+M0J53M9A== 2024-09-17T23:31:30.755 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:31:30.755 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:30.755 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.6 2024-09-17T23:31:30.769 INFO:teuthology.run_tasks:Running task exec... 2024-09-17T23:31:30.777 INFO:teuthology.task.exec:Executing custom commands... 2024-09-17T23:31:30.778 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi192.front.sepia.ceph.com 2024-09-17T23:31:30.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:31.130 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror] 2024-09-17T23:31:31.130 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQABEepmG5txMRAA93pq04i3VLDftm/Lien77g== 2024-09-17T23:31:31.130 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-17T23:31:31.130 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:31.130 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror 2024-09-17T23:31:31.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:31.493 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.0] 2024-09-17T23:31:31.493 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQABEepmolLoNBAAG2PVmEAEvWoGJOQtR4xwBQ== 2024-09-17T23:31:31.493 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:31:31.493 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:31.494 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.0 2024-09-17T23:31:31.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:31.860 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.1] 2024-09-17T23:31:31.861 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQABEepmF8AuOBAAFS1C/BUJujmhMpjfGtgYCw== 2024-09-17T23:31:31.861 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:31:31.861 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:31.861 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.1 2024-09-17T23:31:31.874 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:32.222 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.2] 2024-09-17T23:31:32.222 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQACEepmrwIyAhAAEJdAqu6LjU0l4mw7hxZxVw== 2024-09-17T23:31:32.222 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:31:32.222 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:32.222 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.2 2024-09-17T23:31:32.236 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:32.588 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.3] 2024-09-17T23:31:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQACEepm16i+BxAAXqwkfmKM3gdXcz3R9pnKKg== 2024-09-17T23:31:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:31:32.588 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:32.589 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.3 2024-09-17T23:31:32.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:32.952 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror] 2024-09-17T23:31:32.953 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAqEepmKxtlGxAAr7+r9sND+XuhB5w22FR3BQ== 2024-09-17T23:31:32.953 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-17T23:31:32.953 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:32.953 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror 2024-09-17T23:31:32.966 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:33.315 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.0] 2024-09-17T23:31:33.316 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAqEepmpa0kIRAAk2F66809M64I90UiIlgm+g== 2024-09-17T23:31:33.316 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:31:33.316 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:33.316 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.0 2024-09-17T23:31:33.329 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:33.681 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.1] 2024-09-17T23:31:33.681 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAqEepmc2C8JhAAGi2VcWL798rh/Z5NbMjBFQ== 2024-09-17T23:31:33.682 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:31:33.682 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:33.682 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.1 2024-09-17T23:31:33.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:34.048 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.2] 2024-09-17T23:31:34.048 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAqEepmM+5WLBAA0s0FIZA6HmenUlo0RuLW+A== 2024-09-17T23:31:34.048 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:31:34.048 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:34.049 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.2 2024-09-17T23:31:34.062 DEBUG:teuthology.orchestra.run.smithi192:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-17T23:31:34.410 INFO:teuthology.orchestra.run.smithi192.stdout:[client.mirror.3] 2024-09-17T23:31:34.410 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAqEepmjCvwMRAAGI3QbV2uYET9CBgfpzm4fw== 2024-09-17T23:31:34.410 INFO:teuthology.orchestra.run.smithi192.stdout: caps mon = "profile rbd-mirror" 2024-09-17T23:31:34.410 INFO:teuthology.orchestra.run.smithi192.stdout: caps osd = "profile rbd" 2024-09-17T23:31:34.410 INFO:teuthology.orchestra.run.smithi192.stderr:updated caps for client.mirror.3 2024-09-17T23:31:34.425 INFO:teuthology.run_tasks:Running task workunit... 2024-09-17T23:31:34.438 INFO:tasks.workunit:Pulling workunits from ref b0d356b98e08f6bc59ef15652f0d0e4298fc83c7 2024-09-17T23:31:34.439 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-17T23:31:34.439 DEBUG:teuthology.orchestra.run.smithi192:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-17T23:31:34.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T23:31:34.454 INFO:teuthology.orchestra.run.smithi192.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-17T23:31:34.455 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-17T23:31:34.510 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-17T23:31:34.510 DEBUG:teuthology.orchestra.run.smithi192:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-17T23:31:34.565 INFO:tasks.workunit:timeout=3h 2024-09-17T23:31:34.565 INFO:tasks.workunit:cleanup=True 2024-09-17T23:31:34.565 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout b0d356b98e08f6bc59ef15652f0d0e4298fc83c7 2024-09-17T23:31:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-17T23:32:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Note: switching to 'b0d356b98e08f6bc59ef15652f0d0e4298fc83c7'. 2024-09-17T23:32:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-09-17T23:32:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-17T23:32:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-17T23:32:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:state without impacting any branches by switching back to a branch. 2024-09-17T23:32:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-09-17T23:32:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-17T23:32:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-17T23:32:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-09-17T23:32:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: git switch -c 2024-09-17T23:32:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-09-17T23:32:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Or undo this operation with: 2024-09-17T23:32:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-09-17T23:32:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: git switch - 2024-09-17T23:32:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-09-17T23:32:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-17T23:32:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-09-17T23:32:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:HEAD is now at b0d356b98e0 Merge pull request #59811 from afreen23/wip-68079-squid 2024-09-17T23:32:29.860 DEBUG:teuthology.orchestra.run.smithi192:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-17T23:32:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-17T23:32:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-17T23:32:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-17T23:32:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-17T23:32:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-17T23:32:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-17T23:32:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-17T23:32:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-17T23:32:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-17T23:32:30.754 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-09-17T23:32:30.754 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-17T23:32:30.810 INFO:tasks.workunit:Running workunits matching rbd/compare_mirror_images.sh on cluster1.client.mirror... 2024-09-17T23:32:30.811 INFO:tasks.workunit:Running workunit rbd/compare_mirror_images.sh... 2024-09-17T23:32:30.811 DEBUG:teuthology.orchestra.run.smithi192:workunit test rbd/compare_mirror_images.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b0d356b98e08f6bc59ef15652f0d0e4298fc83c7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=krbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_images.sh 2024-09-17T23:32:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG_PREFIX=image-primary 2024-09-17T23:32:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MIRROR_IMAGE_MODE=snapshot 2024-09-17T23:32:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MIRROR_POOL_MODE=image 2024-09-17T23:32:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT_PREFIX=test-primary 2024-09-17T23:32:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-09-17T23:32:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ RBD_MIRROR_INSTANCES=1 2024-09-17T23:32:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ RBD_MIRROR_MODE=snapshot 2024-09-17T23:32:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ RBD_MIRROR_USE_EXISTING_CLUSTER=1 2024-09-17T23:32:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_images.sh 2024-09-17T23:32:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-17T23:32:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ type xmlstarlet 2024-09-17T23:32:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ XMLSTARLET=xmlstarlet 2024-09-17T23:32:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ RBD_MIRROR_INSTANCES=1 2024-09-17T23:32:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CLUSTER1=cluster1 2024-09-17T23:32:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CLUSTER2=cluster2 2024-09-17T23:32:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-17T23:32:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ POOL=mirror 2024-09-17T23:32:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ PARENT_POOL=mirror_parent 2024-09-17T23:32:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ NS1=ns1 2024-09-17T23:32:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ NS2=ns2 2024-09-17T23:32:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ TEMPDIR= 2024-09-17T23:32:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ID=mirror 2024-09-17T23:32:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-09-17T23:32:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-17T23:32:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ RBD_MIRROR_MODE=snapshot 2024-09-17T23:32:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ MIRROR_POOL_MODE=image 2024-09-17T23:32:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' snapshot = snapshot ']' 2024-09-17T23:32:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ MIRROR_POOL_MODE=image 2024-09-17T23:32:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-17T23:32:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS='--id mirror' 2024-09-17T23:32:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ LAST_MIRROR_INSTANCE=0 2024-09-17T23:32:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_images.sh 2024-09-17T23:32:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-17T23:32:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-17T23:32:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_BIN=. 2024-09-17T23:32:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_SRC=. 2024-09-17T23:32:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-17T23:32:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 -gt 0 ']' 2024-09-17T23:32:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ setup 2024-09-17T23:32:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local c 2024-09-17T23:32:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-17T23:32:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ setup_tempdir 2024-09-17T23:32:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:32:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ mktemp -d 2024-09-17T23:32:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ TEMPDIR=/tmp/tmp.LnAXrU1SbO 2024-09-17T23:32:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -z 1 ']' 2024-09-17T23:32:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ setup_pools cluster1 cluster2 2024-09-17T23:32:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local remote_cluster=cluster2 2024-09-17T23:32:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mon_map_file 2024-09-17T23:32:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mon_addr 2024-09-17T23:32:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local admin_key_file 2024-09-17T23:32:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local uuid 2024-09-17T23:32:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-09-17T23:32:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-17T23:32:31.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror' already exists 2024-09-17T23:32:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-09-17T23:32:31.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-17T23:32:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror_parent' already exists 2024-09-17T23:32:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-09-17T23:32:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-17T23:32:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-09-17T23:32:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-17T23:32:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:32:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-09-17T23:32:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-09-17T23:32:38.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-17T23:32:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-17T23:32:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-17T23:32:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-09-17T23:32:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-17T23:32:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -z ']' 2024-09-17T23:32:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -z ']' 2024-09-17T23:32:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_add cluster1 mirror cluster2 2024-09-17T23:32:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local client_cluster=cluster2 2024-09-17T23:32:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local remote_cluster=cluster2 2024-09-17T23:32:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local uuid_var_name 2024-09-17T23:32:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:32:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local error_code 2024-09-17T23:32:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local peer_uuid 2024-09-17T23:32:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 16 32 2024-09-17T23:32:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-09-17T23:32:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-17T23:32:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_uuid=742b2e65-f023-4619-b4b6-32c6d05bd65f 2024-09-17T23:32:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=0 2024-09-17T23:32:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-09-17T23:32:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 17 ']' 2024-09-17T23:32:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 0 -eq 0 2024-09-17T23:32:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:32:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:32:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-17T23:32:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror_parent 2024-09-17T23:32:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local client_cluster=cluster2 2024-09-17T23:32:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local remote_cluster=cluster2 2024-09-17T23:32:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local uuid_var_name 2024-09-17T23:32:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:32:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local error_code 2024-09-17T23:32:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local peer_uuid 2024-09-17T23:32:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 16 32 2024-09-17T23:32:38.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-09-17T23:32:38.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-17T23:32:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_uuid=64bf0507-5e13-4f65-8902-31f9a147b446 2024-09-17T23:32:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=0 2024-09-17T23:32:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-09-17T23:32:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 17 ']' 2024-09-17T23:32:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 0 -eq 0 2024-09-17T23:32:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:32:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:32:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ setup_pools cluster2 cluster1 2024-09-17T23:32:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:32:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local remote_cluster=cluster1 2024-09-17T23:32:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mon_map_file 2024-09-17T23:32:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mon_addr 2024-09-17T23:32:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local admin_key_file 2024-09-17T23:32:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local uuid 2024-09-17T23:32:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-09-17T23:32:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-17T23:32:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror' already exists 2024-09-17T23:32:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-09-17T23:32:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-17T23:32:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror_parent' already exists 2024-09-17T23:32:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-09-17T23:32:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-17T23:32:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-09-17T23:32:43.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-17T23:32:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:32:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-09-17T23:32:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-09-17T23:32:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-17T23:32:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-17T23:32:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-17T23:32:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-09-17T23:32:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-17T23:32:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -z ']' 2024-09-17T23:32:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -z ']' 2024-09-17T23:32:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_add cluster2 mirror cluster1 2024-09-17T23:32:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:32:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local client_cluster=cluster1 2024-09-17T23:32:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local remote_cluster=cluster1 2024-09-17T23:32:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local uuid_var_name 2024-09-17T23:32:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:32:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local error_code 2024-09-17T23:32:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local peer_uuid 2024-09-17T23:32:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 16 32 2024-09-17T23:32:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-09-17T23:32:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-17T23:32:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_uuid=179d45bc-2d6d-4c24-a5a3-733fa565fa1c 2024-09-17T23:32:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=0 2024-09-17T23:32:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-09-17T23:32:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 17 ']' 2024-09-17T23:32:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 0 -eq 0 2024-09-17T23:32:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:32:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:32:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-17T23:32:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:32:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror_parent 2024-09-17T23:32:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local client_cluster=cluster1 2024-09-17T23:32:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local remote_cluster=cluster1 2024-09-17T23:32:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local uuid_var_name 2024-09-17T23:32:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:32:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local error_code 2024-09-17T23:32:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local peer_uuid 2024-09-17T23:32:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 16 32 2024-09-17T23:32:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-09-17T23:32:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-17T23:32:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ peer_uuid=cea7a473-beb6-4fa6-bf31-5ae093ce0b67 2024-09-17T23:32:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ error_code=0 2024-09-17T23:32:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set -e 2024-09-17T23:32:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 17 ']' 2024-09-17T23:32:47.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test 0 -eq 0 2024-09-17T23:32:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:32:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:32:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:32:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ start_mirrors cluster1 2024-09-17T23:32:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 0 2024-09-17T23:32:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:32:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ start_mirror cluster1:0 2024-09-17T23:32:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-09-17T23:32:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-09-17T23:32:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-17T23:32:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster1:0 2024-09-17T23:32:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-09-17T23:32:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-09-17T23:32:47.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster1 2024-09-17T23:32:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-09-17T23:32:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-17T23:32:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster1 2024-09-17T23:32:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-09-17T23:32:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-09-17T23:32:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-09-17T23:32:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-09-17T23:32:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-17T23:32:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ start_mirrors cluster2 2024-09-17T23:32:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:32:47.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 0 2024-09-17T23:32:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-17T23:32:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ start_mirror cluster2:0 2024-09-17T23:32:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2:0 2024-09-17T23:32:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance 2024-09-17T23:32:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-17T23:32:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local val=cluster2:0 2024-09-17T23:32:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster_var_name=cluster 2024-09-17T23:32:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local instance_var_name=instance 2024-09-17T23:32:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cluster=cluster2 2024-09-17T23:32:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ instance=0 2024-09-17T23:32:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-17T23:32:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval cluster=cluster2 2024-09-17T23:32:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster2 2024-09-17T23:32:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ eval instance=0 2024-09-17T23:32:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-09-17T23:32:47.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-09-17T23:32:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-17T23:32:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wget https://download.ceph.com/qa/linux-5.4.tar.gz 2024-09-17T23:32:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:--2024-09-17 23:32:47-- https://download.ceph.com/qa/linux-5.4.tar.gz 2024-09-17T23:32:47.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-09-17T23:32:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:443... connected. 2024-09-17T23:32:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:HTTP request sent, awaiting response... 200 OK 2024-09-17T23:32:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Length: 172616875 (165M) [application/octet-stream] 2024-09-17T23:32:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:Saving to: ‘linux-5.4.tar.gz’ 2024-09-17T23:32:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-09-17T23:32:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 0K .......... .......... .......... .......... .......... 0% 2.36M 70s 2024-09-17T23:32:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50K .......... .......... .......... .......... .......... 0% 2.38M 69s 2024-09-17T23:32:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 100K .......... .......... .......... .......... .......... 0% 293M 46s 2024-09-17T23:32:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 150K .......... .......... .......... .......... .......... 0% 2.38M 52s 2024-09-17T23:32:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 200K .......... .......... .......... .......... .......... 0% 412M 42s 2024-09-17T23:32:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 250K .......... .......... .......... .......... .......... 0% 444M 35s 2024-09-17T23:32:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 300K .......... .......... .......... .......... .......... 0% 416M 30s 2024-09-17T23:32:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 350K .......... .......... .......... .......... .......... 0% 351M 26s 2024-09-17T23:32:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 400K .......... .......... .......... .......... .......... 0% 2.40M 31s 2024-09-17T23:32:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 450K .......... .......... .......... .......... .......... 0% 391M 28s 2024-09-17T23:32:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 500K .......... .......... .......... .......... .......... 0% 420M 25s 2024-09-17T23:32:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 550K .......... .......... .......... .......... .......... 0% 385M 23s 2024-09-17T23:32:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 600K .......... .......... .......... .......... .......... 0% 445M 21s 2024-09-17T23:32:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 650K .......... .......... .......... .......... .......... 0% 421M 20s 2024-09-17T23:32:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 700K .......... .......... .......... .......... .......... 0% 28.6M 19s 2024-09-17T23:32:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 750K .......... .......... .......... .......... .......... 0% 24.0M 18s 2024-09-17T23:32:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 800K .......... .......... .......... .......... .......... 0% 3.04M 20s 2024-09-17T23:32:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 850K .......... .......... .......... .......... .......... 0% 421M 19s 2024-09-17T23:32:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 900K .......... .......... .......... .......... .......... 0% 386M 18s 2024-09-17T23:32:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 950K .......... .......... .......... .......... .......... 0% 414M 17s 2024-09-17T23:32:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1000K .......... .......... .......... .......... .......... 0% 438M 17s 2024-09-17T23:32:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1050K .......... .......... .......... .......... .......... 0% 422M 16s 2024-09-17T23:32:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1100K .......... .......... .......... .......... .......... 0% 396M 15s 2024-09-17T23:32:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1150K .......... .......... .......... .......... .......... 0% 27.5M 15s 2024-09-17T23:32:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1200K .......... .......... .......... .......... .......... 0% 27.9M 14s 2024-09-17T23:32:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1250K .......... .......... .......... .......... .......... 0% 30.8M 14s 2024-09-17T23:32:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1300K .......... .......... .......... .......... .......... 0% 35.3M 14s 2024-09-17T23:32:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1350K .......... .......... .......... .......... .......... 0% 20.7M 13s 2024-09-17T23:32:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1400K .......... .......... .......... .......... .......... 0% 29.4M 13s 2024-09-17T23:32:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1450K .......... .......... .......... .......... .......... 0% 45.3M 13s 2024-09-17T23:32:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1500K .......... .......... .......... .......... .......... 0% 23.0M 13s 2024-09-17T23:32:47.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1550K .......... .......... .......... .......... .......... 0% 26.4M 12s 2024-09-17T23:32:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1600K .......... .......... .......... .......... .......... 0% 10.9M 13s 2024-09-17T23:32:47.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1650K .......... .......... .......... .......... .......... 1% 397M 12s 2024-09-17T23:32:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1700K .......... .......... .......... .......... .......... 1% 28.8M 12s 2024-09-17T23:32:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1750K .......... .......... .......... .......... .......... 1% 144M 12s 2024-09-17T23:32:47.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1800K .......... .......... .......... .......... .......... 1% 26.5M 12s 2024-09-17T23:32:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1850K .......... .......... .......... .......... .......... 1% 22.4M 11s 2024-09-17T23:32:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1900K .......... .......... .......... .......... .......... 1% 23.8M 11s 2024-09-17T23:32:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 1950K .......... .......... .......... .......... .......... 1% 24.9M 11s 2024-09-17T23:32:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2000K .......... .......... .......... .......... .......... 1% 380M 11s 2024-09-17T23:32:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2050K .......... .......... .......... .......... .......... 1% 22.5M 11s 2024-09-17T23:32:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2100K .......... .......... .......... .......... .......... 1% 24.1M 11s 2024-09-17T23:32:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2150K .......... .......... .......... .......... .......... 1% 54.1M 11s 2024-09-17T23:32:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2200K .......... .......... .......... .......... .......... 1% 15.4M 11s 2024-09-17T23:32:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2250K .......... .......... .......... .......... .......... 1% 424M 10s 2024-09-17T23:32:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2300K .......... .......... .......... .......... .......... 1% 21.8M 10s 2024-09-17T23:32:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2350K .......... .......... .......... .......... .......... 1% 23.7M 10s 2024-09-17T23:32:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2400K .......... .......... .......... .......... .......... 1% 27.0M 10s 2024-09-17T23:32:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2450K .......... .......... .......... .......... .......... 1% 23.4M 10s 2024-09-17T23:32:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2500K .......... .......... .......... .......... .......... 1% 381M 10s 2024-09-17T23:32:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2550K .......... .......... .......... .......... .......... 1% 21.7M 10s 2024-09-17T23:32:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2600K .......... .......... .......... .......... .......... 1% 25.3M 10s 2024-09-17T23:32:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2650K .......... .......... .......... .......... .......... 1% 22.4M 10s 2024-09-17T23:32:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2700K .......... .......... .......... .......... .......... 1% 25.8M 10s 2024-09-17T23:32:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2750K .......... .......... .......... .......... .......... 1% 23.9M 10s 2024-09-17T23:32:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2800K .......... .......... .......... .......... .......... 1% 316M 9s 2024-09-17T23:32:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2850K .......... .......... .......... .......... .......... 1% 23.4M 9s 2024-09-17T23:32:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2900K .......... .......... .......... .......... .......... 1% 25.6M 9s 2024-09-17T23:32:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2950K .......... .......... .......... .......... .......... 1% 25.1M 9s 2024-09-17T23:32:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3000K .......... .......... .......... .......... .......... 1% 402M 9s 2024-09-17T23:32:47.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3050K .......... .......... .......... .......... .......... 1% 22.3M 9s 2024-09-17T23:32:47.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3100K .......... .......... .......... .......... .......... 1% 21.1M 9s 2024-09-17T23:32:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3150K .......... .......... .......... .......... .......... 1% 25.7M 9s 2024-09-17T23:32:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3200K .......... .......... .......... .......... .......... 1% 27.5M 9s 2024-09-17T23:32:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3250K .......... .......... .......... .......... .......... 1% 23.3M 9s 2024-09-17T23:32:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3300K .......... .......... .......... .......... .......... 1% 364M 9s 2024-09-17T23:32:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3350K .......... .......... .......... .......... .......... 2% 20.8M 9s 2024-09-17T23:32:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3400K .......... .......... .......... .......... .......... 2% 22.3M 9s 2024-09-17T23:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3450K .......... .......... .......... .......... .......... 2% 24.6M 9s 2024-09-17T23:32:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3500K .......... .......... .......... .......... .......... 2% 265M 9s 2024-09-17T23:32:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3550K .......... .......... .......... .......... .......... 2% 23.6M 9s 2024-09-17T23:32:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3600K .......... .......... .......... .......... .......... 2% 28.7M 9s 2024-09-17T23:32:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3650K .......... .......... .......... .......... .......... 2% 21.4M 8s 2024-09-17T23:32:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3700K .......... .......... .......... .......... .......... 2% 27.8M 8s 2024-09-17T23:32:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3750K .......... .......... .......... .......... .......... 2% 182M 8s 2024-09-17T23:32:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3800K .......... .......... .......... .......... .......... 2% 23.8M 8s 2024-09-17T23:32:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3850K .......... .......... .......... .......... .......... 2% 28.1M 8s 2024-09-17T23:32:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3900K .......... .......... .......... .......... .......... 2% 23.4M 8s 2024-09-17T23:32:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 3950K .......... .......... .......... .......... .......... 2% 22.2M 8s 2024-09-17T23:32:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4000K .......... .......... .......... .......... .......... 2% 35.5M 8s 2024-09-17T23:32:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4050K .......... .......... .......... .......... .......... 2% 37.7M 8s 2024-09-17T23:32:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4100K .......... .......... .......... .......... .......... 2% 38.1M 8s 2024-09-17T23:32:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4150K .......... .......... .......... .......... .......... 2% 22.1M 8s 2024-09-17T23:32:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4200K .......... .......... .......... .......... .......... 2% 30.7M 8s 2024-09-17T23:32:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4250K .......... .......... .......... .......... .......... 2% 49.4M 8s 2024-09-17T23:32:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4300K .......... .......... .......... .......... .......... 2% 24.0M 8s 2024-09-17T23:32:47.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4350K .......... .......... .......... .......... .......... 2% 23.6M 8s 2024-09-17T23:32:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4400K .......... .......... .......... .......... .......... 2% 31.5M 8s 2024-09-17T23:32:47.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4450K .......... .......... .......... .......... .......... 2% 26.7M 8s 2024-09-17T23:32:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4500K .......... .......... .......... .......... .......... 2% 22.6M 8s 2024-09-17T23:32:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4550K .......... .......... .......... .......... .......... 2% 54.6M 8s 2024-09-17T23:32:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4600K .......... .......... .......... .......... .......... 2% 24.6M 8s 2024-09-17T23:32:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4650K .......... .......... .......... .......... .......... 2% 34.2M 8s 2024-09-17T23:32:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4700K .......... .......... .......... .......... .......... 2% 23.4M 8s 2024-09-17T23:32:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4750K .......... .......... .......... .......... .......... 2% 24.5M 8s 2024-09-17T23:32:47.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4800K .......... .......... .......... .......... .......... 2% 61.7M 8s 2024-09-17T23:32:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4850K .......... .......... .......... .......... .......... 2% 22.6M 8s 2024-09-17T23:32:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4900K .......... .......... .......... .......... .......... 2% 35.5M 8s 2024-09-17T23:32:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 4950K .......... .......... .......... .......... .......... 2% 25.4M 8s 2024-09-17T23:32:47.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5000K .......... .......... .......... .......... .......... 2% 23.3M 8s 2024-09-17T23:32:47.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5050K .......... .......... .......... .......... .......... 3% 55.6M 8s 2024-09-17T23:32:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5100K .......... .......... .......... .......... .......... 3% 33.7M 8s 2024-09-17T23:32:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5150K .......... .......... .......... .......... .......... 3% 18.5M 8s 2024-09-17T23:32:47.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5200K .......... .......... .......... .......... .......... 3% 38.2M 8s 2024-09-17T23:32:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5250K .......... .......... .......... .......... .......... 3% 23.9M 8s 2024-09-17T23:32:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5300K .......... .......... .......... .......... .......... 3% 55.4M 7s 2024-09-17T23:32:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5350K .......... .......... .......... .......... .......... 3% 23.4M 7s 2024-09-17T23:32:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5400K .......... .......... .......... .......... .......... 3% 29.3M 7s 2024-09-17T23:32:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5450K .......... .......... .......... .......... .......... 3% 27.9M 7s 2024-09-17T23:32:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5500K .......... .......... .......... .......... .......... 3% 24.9M 7s 2024-09-17T23:32:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5550K .......... .......... .......... .......... .......... 3% 22.5M 7s 2024-09-17T23:32:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5600K .......... .......... .......... .......... .......... 3% 65.9M 7s 2024-09-17T23:32:47.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5650K .......... .......... .......... .......... .......... 3% 23.3M 7s 2024-09-17T23:32:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5700K .......... .......... .......... .......... .......... 3% 34.6M 7s 2024-09-17T23:32:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5750K .......... .......... .......... .......... .......... 3% 23.9M 7s 2024-09-17T23:32:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5800K .......... .......... .......... .......... .......... 3% 58.0M 7s 2024-09-17T23:32:47.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5850K .......... .......... .......... .......... .......... 3% 23.5M 7s 2024-09-17T23:32:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5900K .......... .......... .......... .......... .......... 3% 37.8M 7s 2024-09-17T23:32:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 5950K .......... .......... .......... .......... .......... 3% 22.7M 7s 2024-09-17T23:32:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6000K .......... .......... .......... .......... .......... 3% 25.3M 7s 2024-09-17T23:32:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6050K .......... .......... .......... .......... .......... 3% 22.7M 7s 2024-09-17T23:32:47.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6100K .......... .......... .......... .......... .......... 3% 60.6M 7s 2024-09-17T23:32:47.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6150K .......... .......... .......... .......... .......... 3% 32.6M 7s 2024-09-17T23:32:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6200K .......... .......... .......... .......... .......... 3% 24.8M 7s 2024-09-17T23:32:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6250K .......... .......... .......... .......... .......... 3% 25.8M 7s 2024-09-17T23:32:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6300K .......... .......... .......... .......... .......... 3% 55.8M 7s 2024-09-17T23:32:47.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6350K .......... .......... .......... .......... .......... 3% 22.9M 7s 2024-09-17T23:32:47.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6400K .......... .......... .......... .......... .......... 3% 23.6M 7s 2024-09-17T23:32:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6450K .......... .......... .......... .......... .......... 3% 38.5M 7s 2024-09-17T23:32:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6500K .......... .......... .......... .......... .......... 3% 24.3M 7s 2024-09-17T23:32:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6550K .......... .......... .......... .......... .......... 3% 48.0M 7s 2024-09-17T23:32:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6600K .......... .......... .......... .......... .......... 3% 29.0M 7s 2024-09-17T23:32:47.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6650K .......... .......... .......... .......... .......... 3% 30.0M 7s 2024-09-17T23:32:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6700K .......... .......... .......... .......... .......... 4% 21.3M 7s 2024-09-17T23:32:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6750K .......... .......... .......... .......... .......... 4% 26.9M 7s 2024-09-17T23:32:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6800K .......... .......... .......... .......... .......... 4% 23.9M 7s 2024-09-17T23:32:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6850K .......... .......... .......... .......... .......... 4% 52.4M 7s 2024-09-17T23:32:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6900K .......... .......... .......... .......... .......... 4% 23.4M 7s 2024-09-17T23:32:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 6950K .......... .......... .......... .......... .......... 4% 51.3M 7s 2024-09-17T23:32:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7000K .......... .......... .......... .......... .......... 4% 21.1M 7s 2024-09-17T23:32:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7050K .......... .......... .......... .......... .......... 4% 48.0M 7s 2024-09-17T23:32:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7100K .......... .......... .......... .......... .......... 4% 23.3M 7s 2024-09-17T23:32:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7150K .......... .......... .......... .......... .......... 4% 25.9M 7s 2024-09-17T23:32:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7200K .......... .......... .......... .......... .......... 4% 35.3M 7s 2024-09-17T23:32:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7250K .......... .......... .......... .......... .......... 4% 23.3M 7s 2024-09-17T23:32:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7300K .......... .......... .......... .......... .......... 4% 24.6M 7s 2024-09-17T23:32:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7350K .......... .......... .......... .......... .......... 4% 59.2M 7s 2024-09-17T23:32:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7400K .......... .......... .......... .......... .......... 4% 23.6M 7s 2024-09-17T23:32:47.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7450K .......... .......... .......... .......... .......... 4% 38.0M 7s 2024-09-17T23:32:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7500K .......... .......... .......... .......... .......... 4% 24.0M 7s 2024-09-17T23:32:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7550K .......... .......... .......... .......... .......... 4% 22.7M 7s 2024-09-17T23:32:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7600K .......... .......... .......... .......... .......... 4% 54.2M 7s 2024-09-17T23:32:47.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7650K .......... .......... .......... .......... .......... 4% 24.2M 7s 2024-09-17T23:32:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7700K .......... .......... .......... .......... .......... 4% 37.7M 7s 2024-09-17T23:32:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7750K .......... .......... .......... .......... .......... 4% 22.3M 7s 2024-09-17T23:32:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7800K .......... .......... .......... .......... .......... 4% 23.0M 7s 2024-09-17T23:32:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7850K .......... .......... .......... .......... .......... 4% 58.8M 7s 2024-09-17T23:32:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7900K .......... .......... .......... .......... .......... 4% 25.4M 7s 2024-09-17T23:32:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 7950K .......... .......... .......... .......... .......... 4% 22.7M 7s 2024-09-17T23:32:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8000K .......... .......... .......... .......... .......... 4% 34.6M 7s 2024-09-17T23:32:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8050K .......... .......... .......... .......... .......... 4% 26.1M 7s 2024-09-17T23:32:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8100K .......... .......... .......... .......... .......... 4% 47.9M 7s 2024-09-17T23:32:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8150K .......... .......... .......... .......... .......... 4% 24.6M 7s 2024-09-17T23:32:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8200K .......... .......... .......... .......... .......... 4% 26.5M 7s 2024-09-17T23:32:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8250K .......... .......... .......... .......... .......... 4% 33.5M 7s 2024-09-17T23:32:47.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8300K .......... .......... .......... .......... .......... 4% 52.7M 7s 2024-09-17T23:32:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8350K .......... .......... .......... .......... .......... 4% 14.9M 7s 2024-09-17T23:32:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8400K .......... .......... .......... .......... .......... 5% 53.4M 7s 2024-09-17T23:32:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8450K .......... .......... .......... .......... .......... 5% 33.3M 7s 2024-09-17T23:32:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8500K .......... .......... .......... .......... .......... 5% 27.0M 7s 2024-09-17T23:32:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8550K .......... .......... .......... .......... .......... 5% 22.1M 7s 2024-09-17T23:32:47.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8600K .......... .......... .......... .......... .......... 5% 49.0M 7s 2024-09-17T23:32:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8650K .......... .......... .......... .......... .......... 5% 26.6M 7s 2024-09-17T23:32:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8700K .......... .......... .......... .......... .......... 5% 44.1M 7s 2024-09-17T23:32:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8750K .......... .......... .......... .......... .......... 5% 21.0M 7s 2024-09-17T23:32:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8800K .......... .......... .......... .......... .......... 5% 23.9M 7s 2024-09-17T23:32:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8850K .......... .......... .......... .......... .......... 5% 49.8M 7s 2024-09-17T23:32:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8900K .......... .......... .......... .......... .......... 5% 26.5M 7s 2024-09-17T23:32:47.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 8950K .......... .......... .......... .......... .......... 5% 31.2M 7s 2024-09-17T23:32:47.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9000K .......... .......... .......... .......... .......... 5% 24.0M 7s 2024-09-17T23:32:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9050K .......... .......... .......... .......... .......... 5% 24.2M 7s 2024-09-17T23:32:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9100K .......... .......... .......... .......... .......... 5% 49.6M 6s 2024-09-17T23:32:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9150K .......... .......... .......... .......... .......... 5% 24.8M 6s 2024-09-17T23:32:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9200K .......... .......... .......... .......... .......... 5% 29.4M 6s 2024-09-17T23:32:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9250K .......... .......... .......... .......... .......... 5% 31.5M 6s 2024-09-17T23:32:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9300K .......... .......... .......... .......... .......... 5% 26.4M 6s 2024-09-17T23:32:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9350K .......... .......... .......... .......... .......... 5% 46.8M 6s 2024-09-17T23:32:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9400K .......... .......... .......... .......... .......... 5% 23.4M 6s 2024-09-17T23:32:48.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9450K .......... .......... .......... .......... .......... 5% 38.3M 6s 2024-09-17T23:32:48.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9500K .......... .......... .......... .......... .......... 5% 23.5M 6s 2024-09-17T23:32:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9550K .......... .......... .......... .......... .......... 5% 22.8M 6s 2024-09-17T23:32:48.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9600K .......... .......... .......... .......... .......... 5% 23.0M 6s 2024-09-17T23:32:48.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9650K .......... .......... .......... .......... .......... 5% 61.7M 6s 2024-09-17T23:32:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9700K .......... .......... .......... .......... .......... 5% 23.1M 6s 2024-09-17T23:32:48.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9750K .......... .......... .......... .......... .......... 5% 50.2M 6s 2024-09-17T23:32:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9800K .......... .......... .......... .......... .......... 5% 20.6M 6s 2024-09-17T23:32:48.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9850K .......... .......... .......... .......... .......... 5% 63.4M 6s 2024-09-17T23:32:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9900K .......... .......... .......... .......... .......... 5% 23.3M 6s 2024-09-17T23:32:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 9950K .......... .......... .......... .......... .......... 5% 23.2M 6s 2024-09-17T23:32:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10000K .......... .......... .......... .......... .......... 5% 37.8M 6s 2024-09-17T23:32:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10050K .......... .......... .......... .......... .......... 5% 21.8M 6s 2024-09-17T23:32:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10100K .......... .......... .......... .......... .......... 6% 24.4M 6s 2024-09-17T23:32:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10150K .......... .......... .......... .......... .......... 6% 104M 6s 2024-09-17T23:32:48.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10200K .......... .......... .......... .......... .......... 6% 30.3M 6s 2024-09-17T23:32:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10250K .......... .......... .......... .......... .......... 6% 23.3M 6s 2024-09-17T23:32:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10300K .......... .......... .......... .......... .......... 6% 23.8M 6s 2024-09-17T23:32:48.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10350K .......... .......... .......... .......... .......... 6% 25.3M 6s 2024-09-17T23:32:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10400K .......... .......... .......... .......... .......... 6% 55.6M 6s 2024-09-17T23:32:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10450K .......... .......... .......... .......... .......... 6% 23.2M 6s 2024-09-17T23:32:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10500K .......... .......... .......... .......... .......... 6% 29.4M 6s 2024-09-17T23:32:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10550K .......... .......... .......... .......... .......... 6% 29.6M 6s 2024-09-17T23:32:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10600K .......... .......... .......... .......... .......... 6% 21.6M 6s 2024-09-17T23:32:48.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10650K .......... .......... .......... .......... .......... 6% 58.9M 6s 2024-09-17T23:32:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10700K .......... .......... .......... .......... .......... 6% 23.4M 6s 2024-09-17T23:32:48.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10750K .......... .......... .......... .......... .......... 6% 34.1M 6s 2024-09-17T23:32:48.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10800K .......... .......... .......... .......... .......... 6% 25.4M 6s 2024-09-17T23:32:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10850K .......... .......... .......... .......... .......... 6% 23.9M 6s 2024-09-17T23:32:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10900K .......... .......... .......... .......... .......... 6% 49.8M 6s 2024-09-17T23:32:48.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 10950K .......... .......... .......... .......... .......... 6% 24.2M 6s 2024-09-17T23:32:48.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11000K .......... .......... .......... .......... .......... 6% 39.4M 6s 2024-09-17T23:32:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11050K .......... .......... .......... .......... .......... 6% 24.4M 6s 2024-09-17T23:32:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11100K .......... .......... .......... .......... .......... 6% 57.3M 6s 2024-09-17T23:32:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11150K .......... .......... .......... .......... .......... 6% 14.7M 6s 2024-09-17T23:32:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11200K .......... .......... .......... .......... .......... 6% 56.8M 6s 2024-09-17T23:32:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11250K .......... .......... .......... .......... .......... 6% 36.9M 6s 2024-09-17T23:32:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11300K .......... .......... .......... .......... .......... 6% 24.0M 6s 2024-09-17T23:32:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11350K .......... .......... .......... .......... .......... 6% 22.6M 6s 2024-09-17T23:32:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11400K .......... .......... .......... .......... .......... 6% 64.8M 6s 2024-09-17T23:32:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11450K .......... .......... .......... .......... .......... 6% 23.5M 6s 2024-09-17T23:32:48.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11500K .......... .......... .......... .......... .......... 6% 37.8M 6s 2024-09-17T23:32:48.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11550K .......... .......... .......... .......... .......... 6% 23.4M 6s 2024-09-17T23:32:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11600K .......... .......... .......... .......... .......... 6% 24.4M 6s 2024-09-17T23:32:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11650K .......... .......... .......... .......... .......... 6% 54.7M 6s 2024-09-17T23:32:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11700K .......... .......... .......... .......... .......... 6% 23.4M 6s 2024-09-17T23:32:48.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11750K .......... .......... .......... .......... .......... 7% 35.8M 6s 2024-09-17T23:32:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11800K .......... .......... .......... .......... .......... 7% 26.8M 6s 2024-09-17T23:32:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11850K .......... .......... .......... .......... .......... 7% 20.6M 6s 2024-09-17T23:32:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11900K .......... .......... .......... .......... .......... 7% 67.1M 6s 2024-09-17T23:32:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 11950K .......... .......... .......... .......... .......... 7% 23.6M 6s 2024-09-17T23:32:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12000K .......... .......... .......... .......... .......... 7% 23.9M 6s 2024-09-17T23:32:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12050K .......... .......... .......... .......... .......... 7% 36.6M 6s 2024-09-17T23:32:48.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12100K .......... .......... .......... .......... .......... 7% 24.8M 6s 2024-09-17T23:32:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12150K .......... .......... .......... .......... .......... 7% 44.2M 6s 2024-09-17T23:32:48.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12200K .......... .......... .......... .......... .......... 7% 26.0M 6s 2024-09-17T23:32:48.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12250K .......... .......... .......... .......... .......... 7% 22.3M 6s 2024-09-17T23:32:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12300K .......... .......... .......... .......... .......... 7% 41.2M 6s 2024-09-17T23:32:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12350K .......... .......... .......... .......... .......... 7% 22.9M 6s 2024-09-17T23:32:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12400K .......... .......... .......... .......... .......... 7% 23.9M 6s 2024-09-17T23:32:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12450K .......... .......... .......... .......... .......... 7% 63.1M 6s 2024-09-17T23:32:48.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12500K .......... .......... .......... .......... .......... 7% 23.6M 6s 2024-09-17T23:32:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12550K .......... .......... .......... .......... .......... 7% 35.8M 6s 2024-09-17T23:32:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12600K .......... .......... .......... .......... .......... 7% 32.6M 6s 2024-09-17T23:32:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12650K .......... .......... .......... .......... .......... 7% 36.5M 6s 2024-09-17T23:32:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12700K .......... .......... .......... .......... .......... 7% 22.5M 6s 2024-09-17T23:32:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12750K .......... .......... .......... .......... .......... 7% 33.0M 6s 2024-09-17T23:32:48.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12800K .......... .......... .......... .......... .......... 7% 26.0M 6s 2024-09-17T23:32:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12850K .......... .......... .......... .......... .......... 7% 22.4M 6s 2024-09-17T23:32:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12900K .......... .......... .......... .......... .......... 7% 24.6M 6s 2024-09-17T23:32:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 12950K .......... .......... .......... .......... .......... 7% 59.0M 6s 2024-09-17T23:32:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13000K .......... .......... .......... .......... .......... 7% 30.0M 6s 2024-09-17T23:32:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13050K .......... .......... .......... .......... .......... 7% 27.7M 6s 2024-09-17T23:32:48.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13100K .......... .......... .......... .......... .......... 7% 23.7M 6s 2024-09-17T23:32:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13150K .......... .......... .......... .......... .......... 7% 23.8M 6s 2024-09-17T23:32:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13200K .......... .......... .......... .......... .......... 7% 50.6M 6s 2024-09-17T23:32:48.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13250K .......... .......... .......... .......... .......... 7% 24.2M 6s 2024-09-17T23:32:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13300K .......... .......... .......... .......... .......... 7% 38.2M 6s 2024-09-17T23:32:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13350K .......... .......... .......... .......... .......... 7% 22.7M 6s 2024-09-17T23:32:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13400K .......... .......... .......... .......... .......... 7% 58.3M 6s 2024-09-17T23:32:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13450K .......... .......... .......... .......... .......... 8% 24.3M 6s 2024-09-17T23:32:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13500K .......... .......... .......... .......... .......... 8% 36.1M 6s 2024-09-17T23:32:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13550K .......... .......... .......... .......... .......... 8% 23.7M 6s 2024-09-17T23:32:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13600K .......... .......... .......... .......... .......... 8% 22.6M 6s 2024-09-17T23:32:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13650K .......... .......... .......... .......... .......... 8% 24.3M 6s 2024-09-17T23:32:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13700K .......... .......... .......... .......... .......... 8% 49.4M 6s 2024-09-17T23:32:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13750K .......... .......... .......... .......... .......... 8% 25.8M 6s 2024-09-17T23:32:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13800K .......... .......... .......... .......... .......... 8% 38.3M 6s 2024-09-17T23:32:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13850K .......... .......... .......... .......... .......... 8% 23.7M 6s 2024-09-17T23:32:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13900K .......... .......... .......... .......... .......... 8% 55.3M 6s 2024-09-17T23:32:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 13950K .......... .......... .......... .......... .......... 8% 22.4M 6s 2024-09-17T23:32:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14000K .......... .......... .......... .......... .......... 8% 25.0M 6s 2024-09-17T23:32:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14050K .......... .......... .......... .......... .......... 8% 37.7M 6s 2024-09-17T23:32:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14100K .......... .......... .......... .......... .......... 8% 23.9M 6s 2024-09-17T23:32:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14150K .......... .......... .......... .......... .......... 8% 23.0M 6s 2024-09-17T23:32:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14200K .......... .......... .......... .......... .......... 8% 47.5M 6s 2024-09-17T23:32:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14250K .......... .......... .......... .......... .......... 8% 29.0M 6s 2024-09-17T23:32:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14300K .......... .......... .......... .......... .......... 8% 31.8M 6s 2024-09-17T23:32:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14350K .......... .......... .......... .......... .......... 8% 24.4M 6s 2024-09-17T23:32:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14400K .......... .......... .......... .......... .......... 8% 26.4M 6s 2024-09-17T23:32:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14450K .......... .......... .......... .......... .......... 8% 43.8M 6s 2024-09-17T23:32:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14500K .......... .......... .......... .......... .......... 8% 23.9M 6s 2024-09-17T23:32:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14550K .......... .......... .......... .......... .......... 8% 35.5M 6s 2024-09-17T23:32:48.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14600K .......... .......... .......... .......... .......... 8% 23.0M 6s 2024-09-17T23:32:48.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14650K .......... .......... .......... .......... .......... 8% 23.4M 6s 2024-09-17T23:32:48.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14700K .......... .......... .......... .......... .......... 8% 66.9M 6s 2024-09-17T23:32:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14750K .......... .......... .......... .......... .......... 8% 22.5M 6s 2024-09-17T23:32:48.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14800K .......... .......... .......... .......... .......... 8% 24.7M 6s 2024-09-17T23:32:48.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14850K .......... .......... .......... .......... .......... 8% 37.8M 6s 2024-09-17T23:32:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14900K .......... .......... .......... .......... .......... 8% 22.0M 6s 2024-09-17T23:32:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 14950K .......... .......... .......... .......... .......... 8% 64.7M 6s 2024-09-17T23:32:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15000K .......... .......... .......... .......... .......... 8% 24.0M 6s 2024-09-17T23:32:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15050K .......... .......... .......... .......... .......... 8% 36.5M 6s 2024-09-17T23:32:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15100K .......... .......... .......... .......... .......... 8% 24.8M 6s 2024-09-17T23:32:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15150K .......... .......... .......... .......... .......... 9% 22.4M 6s 2024-09-17T23:32:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15200K .......... .......... .......... .......... .......... 9% 25.1M 6s 2024-09-17T23:32:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15250K .......... .......... .......... .......... .......... 9% 55.7M 6s 2024-09-17T23:32:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15300K .......... .......... .......... .......... .......... 9% 23.5M 6s 2024-09-17T23:32:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15350K .......... .......... .......... .......... .......... 9% 32.9M 6s 2024-09-17T23:32:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15400K .......... .......... .......... .......... .......... 9% 26.1M 6s 2024-09-17T23:32:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15450K .......... .......... .......... .......... .......... 9% 56.1M 6s 2024-09-17T23:32:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15500K .......... .......... .......... .......... .......... 9% 22.3M 6s 2024-09-17T23:32:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15550K .......... .......... .......... .......... .......... 9% 24.5M 6s 2024-09-17T23:32:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15600K .......... .......... .......... .......... .......... 9% 37.9M 6s 2024-09-17T23:32:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15650K .......... .......... .......... .......... .......... 9% 23.2M 6s 2024-09-17T23:32:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15700K .......... .......... .......... .......... .......... 9% 24.4M 6s 2024-09-17T23:32:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15750K .......... .......... .......... .......... .......... 9% 52.9M 6s 2024-09-17T23:32:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15800K .......... .......... .......... .......... .......... 9% 35.7M 6s 2024-09-17T23:32:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15850K .......... .......... .......... .......... .......... 9% 25.7M 6s 2024-09-17T23:32:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15900K .......... .......... .......... .......... .......... 9% 22.8M 6s 2024-09-17T23:32:48.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 15950K .......... .......... .......... .......... .......... 9% 23.4M 6s 2024-09-17T23:32:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16000K .......... .......... .......... .......... .......... 9% 45.0M 6s 2024-09-17T23:32:48.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16050K .......... .......... .......... .......... .......... 9% 24.6M 6s 2024-09-17T23:32:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16100K .......... .......... .......... .......... .......... 9% 23.3M 6s 2024-09-17T23:32:48.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16150K .......... .......... .......... .......... .......... 9% 37.7M 6s 2024-09-17T23:32:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16200K .......... .......... .......... .......... .......... 9% 54.1M 6s 2024-09-17T23:32:48.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16250K .......... .......... .......... .......... .......... 9% 25.1M 6s 2024-09-17T23:32:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16300K .......... .......... .......... .......... .......... 9% 35.6M 6s 2024-09-17T23:32:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16350K .......... .......... .......... .......... .......... 9% 17.7M 6s 2024-09-17T23:32:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16400K .......... .......... .......... .......... .......... 9% 37.9M 6s 2024-09-17T23:32:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16450K .......... .......... .......... .......... .......... 9% 23.1M 6s 2024-09-17T23:32:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16500K .......... .......... .......... .......... .......... 9% 57.0M 6s 2024-09-17T23:32:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16550K .......... .......... .......... .......... .......... 9% 25.4M 6s 2024-09-17T23:32:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16600K .......... .......... .......... .......... .......... 9% 35.5M 6s 2024-09-17T23:32:48.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16650K .......... .......... .......... .......... .......... 9% 22.9M 6s 2024-09-17T23:32:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16700K .......... .......... .......... .......... .......... 9% 51.0M 6s 2024-09-17T23:32:48.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16750K .......... .......... .......... .......... .......... 9% 23.8M 6s 2024-09-17T23:32:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16800K .......... .......... .......... .......... .......... 9% 23.0M 6s 2024-09-17T23:32:48.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16850K .......... .......... .......... .......... .......... 10% 38.7M 6s 2024-09-17T23:32:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16900K .......... .......... .......... .......... .......... 10% 22.8M 6s 2024-09-17T23:32:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 16950K .......... .......... .......... .......... .......... 10% 25.0M 6s 2024-09-17T23:32:48.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17000K .......... .......... .......... .......... .......... 10% 58.4M 6s 2024-09-17T23:32:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17050K .......... .......... .......... .......... .......... 10% 23.1M 6s 2024-09-17T23:32:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17100K .......... .......... .......... .......... .......... 10% 46.4M 6s 2024-09-17T23:32:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17150K .......... .......... .......... .......... .......... 10% 21.3M 6s 2024-09-17T23:32:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17200K .......... .......... .......... .......... .......... 10% 22.8M 6s 2024-09-17T23:32:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17250K .......... .......... .......... .......... .......... 10% 48.1M 6s 2024-09-17T23:32:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17300K .......... .......... .......... .......... .......... 10% 28.2M 6s 2024-09-17T23:32:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17350K .......... .......... .......... .......... .......... 10% 33.4M 6s 2024-09-17T23:32:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17400K .......... .......... .......... .......... .......... 10% 20.9M 6s 2024-09-17T23:32:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17450K .......... .......... .......... .......... .......... 10% 23.9M 6s 2024-09-17T23:32:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17500K .......... .......... .......... .......... .......... 10% 73.1M 6s 2024-09-17T23:32:48.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17550K .......... .......... .......... .......... .......... 10% 23.0M 6s 2024-09-17T23:32:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17600K .......... .......... .......... .......... .......... 10% 23.9M 6s 2024-09-17T23:32:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17650K .......... .......... .......... .......... .......... 10% 36.7M 6s 2024-09-17T23:32:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17700K .......... .......... .......... .......... .......... 10% 24.9M 6s 2024-09-17T23:32:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17750K .......... .......... .......... .......... .......... 10% 55.6M 6s 2024-09-17T23:32:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17800K .......... .......... .......... .......... .......... 10% 23.7M 6s 2024-09-17T23:32:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17850K .......... .......... .......... .......... .......... 10% 36.9M 6s 2024-09-17T23:32:48.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17900K .......... .......... .......... .......... .......... 10% 23.4M 6s 2024-09-17T23:32:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 17950K .......... .......... .......... .......... .......... 10% 22.5M 6s 2024-09-17T23:32:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18000K .......... .......... .......... .......... .......... 10% 24.7M 6s 2024-09-17T23:32:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18050K .......... .......... .......... .......... .......... 10% 53.0M 6s 2024-09-17T23:32:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18100K .......... .......... .......... .......... .......... 10% 38.3M 6s 2024-09-17T23:32:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18150K .......... .......... .......... .......... .......... 10% 22.6M 6s 2024-09-17T23:32:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18200K .......... .......... .......... .......... .......... 10% 24.2M 6s 2024-09-17T23:32:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18250K .......... .......... .......... .......... .......... 10% 56.7M 6s 2024-09-17T23:32:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18300K .......... .......... .......... .......... .......... 10% 24.8M 6s 2024-09-17T23:32:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18350K .......... .......... .......... .......... .......... 10% 25.4M 6s 2024-09-17T23:32:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18400K .......... .......... .......... .......... .......... 10% 35.7M 6s 2024-09-17T23:32:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18450K .......... .......... .......... .......... .......... 10% 24.2M 6s 2024-09-17T23:32:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18500K .......... .......... .......... .......... .......... 11% 22.4M 6s 2024-09-17T23:32:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18550K .......... .......... .......... .......... .......... 11% 56.6M 6s 2024-09-17T23:32:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18600K .......... .......... .......... .......... .......... 11% 22.3M 6s 2024-09-17T23:32:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18650K .......... .......... .......... .......... .......... 11% 27.2M 6s 2024-09-17T23:32:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18700K .......... .......... .......... .......... .......... 11% 33.1M 6s 2024-09-17T23:32:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18750K .......... .......... .......... .......... .......... 11% 31.2M 6s 2024-09-17T23:32:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18800K .......... .......... .......... .......... .......... 11% 33.7M 6s 2024-09-17T23:32:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18850K .......... .......... .......... .......... .......... 11% 23.8M 6s 2024-09-17T23:32:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18900K .......... .......... .......... .......... .......... 11% 37.9M 6s 2024-09-17T23:32:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 18950K .......... .......... .......... .......... .......... 11% 23.5M 6s 2024-09-17T23:32:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19000K .......... .......... .......... .......... .......... 11% 58.8M 6s 2024-09-17T23:32:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19050K .......... .......... .......... .......... .......... 11% 24.1M 6s 2024-09-17T23:32:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19100K .......... .......... .......... .......... .......... 11% 37.7M 6s 2024-09-17T23:32:48.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19150K .......... .......... .......... .......... .......... 11% 23.4M 6s 2024-09-17T23:32:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19200K .......... .......... .......... .......... .......... 11% 21.6M 6s 2024-09-17T23:32:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19250K .......... .......... .......... .......... .......... 11% 25.7M 6s 2024-09-17T23:32:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19300K .......... .......... .......... .......... .......... 11% 55.2M 6s 2024-09-17T23:32:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19350K .......... .......... .......... .......... .......... 11% 24.4M 6s 2024-09-17T23:32:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19400K .......... .......... .......... .......... .......... 11% 31.2M 6s 2024-09-17T23:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19450K .......... .......... .......... .......... .......... 11% 26.1M 6s 2024-09-17T23:32:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19500K .......... .......... .......... .......... .......... 11% 62.2M 6s 2024-09-17T23:32:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19550K .......... .......... .......... .......... .......... 11% 23.7M 6s 2024-09-17T23:32:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19600K .......... .......... .......... .......... .......... 11% 22.6M 6s 2024-09-17T23:32:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19650K .......... .......... .......... .......... .......... 11% 37.2M 5s 2024-09-17T23:32:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19700K .......... .......... .......... .......... .......... 11% 22.7M 5s 2024-09-17T23:32:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19750K .......... .......... .......... .......... .......... 11% 27.4M 5s 2024-09-17T23:32:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19800K .......... .......... .......... .......... .......... 11% 48.3M 5s 2024-09-17T23:32:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19850K .......... .......... .......... .......... .......... 11% 23.1M 5s 2024-09-17T23:32:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19900K .......... .......... .......... .......... .......... 11% 24.8M 5s 2024-09-17T23:32:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 19950K .......... .......... .......... .......... .......... 11% 24.1M 5s 2024-09-17T23:32:48.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20000K .......... .......... .......... .......... .......... 11% 36.9M 5s 2024-09-17T23:32:48.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20050K .......... .......... .......... .......... .......... 11% 53.0M 5s 2024-09-17T23:32:48.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20100K .......... .......... .......... .......... .......... 11% 21.4M 5s 2024-09-17T23:32:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20150K .......... .......... .......... .......... .......... 11% 46.7M 5s 2024-09-17T23:32:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20200K .......... .......... .......... .......... .......... 12% 22.5M 5s 2024-09-17T23:32:48.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20250K .......... .......... .......... .......... .......... 12% 26.0M 5s 2024-09-17T23:32:48.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20300K .......... .......... .......... .......... .......... 12% 41.5M 5s 2024-09-17T23:32:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20350K .......... .......... .......... .......... .......... 12% 29.3M 5s 2024-09-17T23:32:48.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20400K .......... .......... .......... .......... .......... 12% 21.4M 5s 2024-09-17T23:32:48.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20450K .......... .......... .......... .......... .......... 12% 37.9M 5s 2024-09-17T23:32:48.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20500K .......... .......... .......... .......... .......... 12% 24.0M 5s 2024-09-17T23:32:48.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20550K .......... .......... .......... .......... .......... 12% 53.4M 5s 2024-09-17T23:32:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20600K .......... .......... .......... .......... .......... 12% 24.8M 5s 2024-09-17T23:32:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20650K .......... .......... .......... .......... .......... 12% 30.9M 5s 2024-09-17T23:32:48.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20700K .......... .......... .......... .......... .......... 12% 27.3M 5s 2024-09-17T23:32:48.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20750K .......... .......... .......... .......... .......... 12% 23.0M 5s 2024-09-17T23:32:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20800K .......... .......... .......... .......... .......... 12% 22.7M 5s 2024-09-17T23:32:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20850K .......... .......... .......... .......... .......... 12% 59.0M 5s 2024-09-17T23:32:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20900K .......... .......... .......... .......... .......... 12% 37.0M 5s 2024-09-17T23:32:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 20950K .......... .......... .......... .......... .......... 12% 23.4M 5s 2024-09-17T23:32:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21000K .......... .......... .......... .......... .......... 12% 24.4M 5s 2024-09-17T23:32:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21050K .......... .......... .......... .......... .......... 12% 53.2M 5s 2024-09-17T23:32:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21100K .......... .......... .......... .......... .......... 12% 23.8M 5s 2024-09-17T23:32:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21150K .......... .......... .......... .......... .......... 12% 24.7M 5s 2024-09-17T23:32:48.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21200K .......... .......... .......... .......... .......... 12% 33.5M 5s 2024-09-17T23:32:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21250K .......... .......... .......... .......... .......... 12% 24.7M 5s 2024-09-17T23:32:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21300K .......... .......... .......... .......... .......... 12% 47.4M 5s 2024-09-17T23:32:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21350K .......... .......... .......... .......... .......... 12% 25.6M 5s 2024-09-17T23:32:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21400K .......... .......... .......... .......... .......... 12% 35.1M 5s 2024-09-17T23:32:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21450K .......... .......... .......... .......... .......... 12% 22.8M 5s 2024-09-17T23:32:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21500K .......... .......... .......... .......... .......... 12% 24.8M 5s 2024-09-17T23:32:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21550K .......... .......... .......... .......... .......... 12% 23.4M 5s 2024-09-17T23:32:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21600K .......... .......... .......... .......... .......... 12% 56.7M 5s 2024-09-17T23:32:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21650K .......... .......... .......... .......... .......... 12% 23.8M 5s 2024-09-17T23:32:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21700K .......... .......... .......... .......... .......... 12% 38.2M 5s 2024-09-17T23:32:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21750K .......... .......... .......... .......... .......... 12% 22.8M 5s 2024-09-17T23:32:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21800K .......... .......... .......... .......... .......... 12% 51.3M 5s 2024-09-17T23:32:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21850K .......... .......... .......... .......... .......... 12% 25.7M 5s 2024-09-17T23:32:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21900K .......... .......... .......... .......... .......... 13% 36.7M 5s 2024-09-17T23:32:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 21950K .......... .......... .......... .......... .......... 13% 22.3M 5s 2024-09-17T23:32:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22000K .......... .......... .......... .......... .......... 13% 22.9M 5s 2024-09-17T23:32:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22050K .......... .......... .......... .......... .......... 13% 26.7M 5s 2024-09-17T23:32:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22100K .......... .......... .......... .......... .......... 13% 54.9M 5s 2024-09-17T23:32:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22150K .......... .......... .......... .......... .......... 13% 24.1M 5s 2024-09-17T23:32:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22200K .......... .......... .......... .......... .......... 13% 37.3M 5s 2024-09-17T23:32:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22250K .......... .......... .......... .......... .......... 13% 22.9M 5s 2024-09-17T23:32:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22300K .......... .......... .......... .......... .......... 13% 62.2M 5s 2024-09-17T23:32:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22350K .......... .......... .......... .......... .......... 13% 22.9M 5s 2024-09-17T23:32:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22400K .......... .......... .......... .......... .......... 13% 22.6M 5s 2024-09-17T23:32:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22450K .......... .......... .......... .......... .......... 13% 39.0M 5s 2024-09-17T23:32:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22500K .......... .......... .......... .......... .......... 13% 23.6M 5s 2024-09-17T23:32:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22550K .......... .......... .......... .......... .......... 13% 23.6M 5s 2024-09-17T23:32:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22600K .......... .......... .......... .......... .......... 13% 67.3M 5s 2024-09-17T23:32:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22650K .......... .......... .......... .......... .......... 13% 32.1M 5s 2024-09-17T23:32:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22700K .......... .......... .......... .......... .......... 13% 24.7M 5s 2024-09-17T23:32:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22750K .......... .......... .......... .......... .......... 13% 24.4M 5s 2024-09-17T23:32:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22800K .......... .......... .......... .......... .......... 13% 589K 6s 2024-09-17T23:32:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22850K .......... .......... .......... .......... .......... 13% 420M 6s 2024-09-17T23:32:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22900K .......... .......... .......... .......... .......... 13% 423M 6s 2024-09-17T23:32:48.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 22950K .......... .......... .......... .......... .......... 13% 367M 6s 2024-09-17T23:32:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23000K .......... .......... .......... .......... .......... 13% 420M 6s 2024-09-17T23:32:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23050K .......... .......... .......... .......... .......... 13% 424M 6s 2024-09-17T23:32:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23100K .......... .......... .......... .......... .......... 13% 436M 6s 2024-09-17T23:32:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23150K .......... .......... .......... .......... .......... 13% 364M 6s 2024-09-17T23:32:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23200K .......... .......... .......... .......... .......... 13% 369M 6s 2024-09-17T23:32:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23250K .......... .......... .......... .......... .......... 13% 431M 6s 2024-09-17T23:32:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23300K .......... .......... .......... .......... .......... 13% 437M 6s 2024-09-17T23:32:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23350K .......... .......... .......... .......... .......... 13% 377M 6s 2024-09-17T23:32:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23400K .......... .......... .......... .......... .......... 13% 438M 6s 2024-09-17T23:32:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23450K .......... .......... .......... .......... .......... 13% 439M 6s 2024-09-17T23:32:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23500K .......... .......... .......... .......... .......... 13% 437M 6s 2024-09-17T23:32:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23550K .......... .......... .......... .......... .......... 14% 2.58M 6s 2024-09-17T23:32:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23600K .......... .......... .......... .......... .......... 14% 363M 6s 2024-09-17T23:32:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23650K .......... .......... .......... .......... .......... 14% 423M 6s 2024-09-17T23:32:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23700K .......... .......... .......... .......... .......... 14% 422M 6s 2024-09-17T23:32:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23750K .......... .......... .......... .......... .......... 14% 388M 6s 2024-09-17T23:32:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23800K .......... .......... .......... .......... .......... 14% 420M 6s 2024-09-17T23:32:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23850K .......... .......... .......... .......... .......... 14% 415M 6s 2024-09-17T23:32:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23900K .......... .......... .......... .......... .......... 14% 389M 6s 2024-09-17T23:32:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 23950K .......... .......... .......... .......... .......... 14% 352M 6s 2024-09-17T23:32:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24000K .......... .......... .......... .......... .......... 14% 424M 6s 2024-09-17T23:32:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24050K .......... .......... .......... .......... .......... 14% 418M 6s 2024-09-17T23:32:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24100K .......... .......... .......... .......... .......... 14% 6.37M 6s 2024-09-17T23:32:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24150K .......... .......... .......... .......... .......... 14% 363M 6s 2024-09-17T23:32:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24200K .......... .......... .......... .......... .......... 14% 428M 6s 2024-09-17T23:32:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24250K .......... .......... .......... .......... .......... 14% 392M 6s 2024-09-17T23:32:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24300K .......... .......... .......... .......... .......... 14% 404M 6s 2024-09-17T23:32:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24350K .......... .......... .......... .......... .......... 14% 350M 6s 2024-09-17T23:32:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24400K .......... .......... .......... .......... .......... 14% 424M 6s 2024-09-17T23:32:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24450K .......... .......... .......... .......... .......... 14% 407M 6s 2024-09-17T23:32:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24500K .......... .......... .......... .......... .......... 14% 429M 6s 2024-09-17T23:32:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24550K .......... .......... .......... .......... .......... 14% 381M 6s 2024-09-17T23:32:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24600K .......... .......... .......... .......... .......... 14% 390M 6s 2024-09-17T23:32:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24650K .......... .......... .......... .......... .......... 14% 431M 6s 2024-09-17T23:32:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24700K .......... .......... .......... .......... .......... 14% 386M 6s 2024-09-17T23:32:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24750K .......... .......... .......... .......... .......... 14% 349M 5s 2024-09-17T23:32:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24800K .......... .......... .......... .......... .......... 14% 430M 5s 2024-09-17T23:32:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24850K .......... .......... .......... .......... .......... 14% 15.0M 5s 2024-09-17T23:32:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24900K .......... .......... .......... .......... .......... 14% 327M 5s 2024-09-17T23:32:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 24950K .......... .......... .......... .......... .......... 14% 365M 5s 2024-09-17T23:32:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25000K .......... .......... .......... .......... .......... 14% 431M 5s 2024-09-17T23:32:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25050K .......... .......... .......... .......... .......... 14% 431M 5s 2024-09-17T23:32:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25100K .......... .......... .......... .......... .......... 14% 432M 5s 2024-09-17T23:32:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25150K .......... .......... .......... .......... .......... 14% 359M 5s 2024-09-17T23:32:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25200K .......... .......... .......... .......... .......... 14% 399M 5s 2024-09-17T23:32:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25250K .......... .......... .......... .......... .......... 15% 435M 5s 2024-09-17T23:32:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25300K .......... .......... .......... .......... .......... 15% 437M 5s 2024-09-17T23:32:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25350K .......... .......... .......... .......... .......... 15% 391M 5s 2024-09-17T23:32:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25400K .......... .......... .......... .......... .......... 15% 428M 5s 2024-09-17T23:32:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25450K .......... .......... .......... .......... .......... 15% 430M 5s 2024-09-17T23:32:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25500K .......... .......... .......... .......... .......... 15% 437M 5s 2024-09-17T23:32:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25550K .......... .......... .......... .......... .......... 15% 363M 5s 2024-09-17T23:32:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25600K .......... .......... .......... .......... .......... 15% 434M 5s 2024-09-17T23:32:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25650K .......... .......... .......... .......... .......... 15% 419M 5s 2024-09-17T23:32:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25700K .......... .......... .......... .......... .......... 15% 430M 5s 2024-09-17T23:32:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25750K .......... .......... .......... .......... .......... 15% 390M 5s 2024-09-17T23:32:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25800K .......... .......... .......... .......... .......... 15% 443M 5s 2024-09-17T23:32:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25850K .......... .......... .......... .......... .......... 15% 434M 5s 2024-09-17T23:32:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25900K .......... .......... .......... .......... .......... 15% 438M 5s 2024-09-17T23:32:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 25950K .......... .......... .......... .......... .......... 15% 325M 5s 2024-09-17T23:32:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26000K .......... .......... .......... .......... .......... 15% 412M 5s 2024-09-17T23:32:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26050K .......... .......... .......... .......... .......... 15% 423M 5s 2024-09-17T23:32:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26100K .......... .......... .......... .......... .......... 15% 438M 5s 2024-09-17T23:32:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26150K .......... .......... .......... .......... .......... 15% 394M 5s 2024-09-17T23:32:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26200K .......... .......... .......... .......... .......... 15% 443M 5s 2024-09-17T23:32:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26250K .......... .......... .......... .......... .......... 15% 432M 5s 2024-09-17T23:32:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26300K .......... .......... .......... .......... .......... 15% 450M 5s 2024-09-17T23:32:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26350K .......... .......... .......... .......... .......... 15% 10.8M 5s 2024-09-17T23:32:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26400K .......... .......... .......... .......... .......... 15% 141M 5s 2024-09-17T23:32:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26450K .......... .......... .......... .......... .......... 15% 24.3M 5s 2024-09-17T23:32:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26500K .......... .......... .......... .......... .......... 15% 327M 5s 2024-09-17T23:32:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26550K .......... .......... .......... .......... .......... 15% 15.9M 5s 2024-09-17T23:32:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26600K .......... .......... .......... .......... .......... 15% 26.3M 5s 2024-09-17T23:32:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26650K .......... .......... .......... .......... .......... 15% 398M 5s 2024-09-17T23:32:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26700K .......... .......... .......... .......... .......... 15% 449M 5s 2024-09-17T23:32:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26750K .......... .......... .......... .......... .......... 15% 356M 5s 2024-09-17T23:32:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26800K .......... .......... .......... .......... .......... 15% 18.2M 5s 2024-09-17T23:32:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26850K .......... .......... .......... .......... .......... 15% 28.9M 5s 2024-09-17T23:32:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26900K .......... .......... .......... .......... .......... 15% 434M 5s 2024-09-17T23:32:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 26950K .......... .......... .......... .......... .......... 16% 30.4M 5s 2024-09-17T23:32:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27000K .......... .......... .......... .......... .......... 16% 43.3M 5s 2024-09-17T23:32:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27050K .......... .......... .......... .......... .......... 16% 23.6M 5s 2024-09-17T23:32:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27100K .......... .......... .......... .......... .......... 16% 48.5M 5s 2024-09-17T23:32:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27150K .......... .......... .......... .......... .......... 16% 23.1M 5s 2024-09-17T23:32:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27200K .......... .......... .......... .......... .......... 16% 24.6M 5s 2024-09-17T23:32:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27250K .......... .......... .......... .......... .......... 16% 23.2M 5s 2024-09-17T23:32:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27300K .......... .......... .......... .......... .......... 16% 39.3M 5s 2024-09-17T23:32:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27350K .......... .......... .......... .......... .......... 16% 22.7M 5s 2024-09-17T23:32:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27400K .......... .......... .......... .......... .......... 16% 59.1M 5s 2024-09-17T23:32:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27450K .......... .......... .......... .......... .......... 16% 22.5M 5s 2024-09-17T23:32:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27500K .......... .......... .......... .......... .......... 16% 34.5M 5s 2024-09-17T23:32:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27550K .......... .......... .......... .......... .......... 16% 25.3M 5s 2024-09-17T23:32:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27600K .......... .......... .......... .......... .......... 16% 24.8M 5s 2024-09-17T23:32:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27650K .......... .......... .......... .......... .......... 16% 45.3M 5s 2024-09-17T23:32:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27700K .......... .......... .......... .......... .......... 16% 25.0M 5s 2024-09-17T23:32:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27750K .......... .......... .......... .......... .......... 16% 34.7M 5s 2024-09-17T23:32:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27800K .......... .......... .......... .......... .......... 16% 22.9M 5s 2024-09-17T23:32:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27850K .......... .......... .......... .......... .......... 16% 24.8M 5s 2024-09-17T23:32:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27900K .......... .......... .......... .......... .......... 16% 64.7M 5s 2024-09-17T23:32:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 27950K .......... .......... .......... .......... .......... 16% 24.6M 5s 2024-09-17T23:32:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28000K .......... .......... .......... .......... .......... 16% 23.0M 5s 2024-09-17T23:32:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28050K .......... .......... .......... .......... .......... 16% 35.0M 5s 2024-09-17T23:32:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28100K .......... .......... .......... .......... .......... 16% 45.3M 5s 2024-09-17T23:32:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28150K .......... .......... .......... .......... .......... 16% 27.9M 5s 2024-09-17T23:32:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28200K .......... .......... .......... .......... .......... 16% 30.5M 5s 2024-09-17T23:32:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28250K .......... .......... .......... .......... .......... 16% 20.1M 5s 2024-09-17T23:32:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28300K .......... .......... .......... .......... .......... 16% 25.9M 5s 2024-09-17T23:32:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28350K .......... .......... .......... .......... .......... 16% 20.1M 5s 2024-09-17T23:32:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28400K .......... .......... .......... .......... .......... 16% 23.1M 5s 2024-09-17T23:32:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28450K .......... .......... .......... .......... .......... 16% 383M 5s 2024-09-17T23:32:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28500K .......... .......... .......... .......... .......... 16% 24.4M 5s 2024-09-17T23:32:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28550K .......... .......... .......... .......... .......... 16% 21.7M 5s 2024-09-17T23:32:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28600K .......... .......... .......... .......... .......... 16% 26.2M 5s 2024-09-17T23:32:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28650K .......... .......... .......... .......... .......... 17% 359M 5s 2024-09-17T23:32:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28700K .......... .......... .......... .......... .......... 17% 23.5M 5s 2024-09-17T23:32:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28750K .......... .......... .......... .......... .......... 17% 24.7M 5s 2024-09-17T23:32:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28800K .......... .......... .......... .......... .......... 17% 31.7M 5s 2024-09-17T23:32:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28850K .......... .......... .......... .......... .......... 17% 21.5M 5s 2024-09-17T23:32:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28900K .......... .......... .......... .......... .......... 17% 23.8M 5s 2024-09-17T23:32:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 28950K .......... .......... .......... .......... .......... 17% 320M 5s 2024-09-17T23:32:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29000K .......... .......... .......... .......... .......... 17% 23.6M 5s 2024-09-17T23:32:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29050K .......... .......... .......... .......... .......... 17% 23.5M 5s 2024-09-17T23:32:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29100K .......... .......... .......... .......... .......... 17% 23.7M 5s 2024-09-17T23:32:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29150K .......... .......... .......... .......... .......... 17% 21.7M 5s 2024-09-17T23:32:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29200K .......... .......... .......... .......... .......... 17% 258M 5s 2024-09-17T23:32:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29250K .......... .......... .......... .......... .......... 17% 27.4M 5s 2024-09-17T23:32:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29300K .......... .......... .......... .......... .......... 17% 22.6M 5s 2024-09-17T23:32:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29350K .......... .......... .......... .......... .......... 17% 48.1M 5s 2024-09-17T23:32:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29400K .......... .......... .......... .......... .......... 17% 31.3M 5s 2024-09-17T23:32:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29450K .......... .......... .......... .......... .......... 17% 29.1M 5s 2024-09-17T23:32:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29500K .......... .......... .......... .......... .......... 17% 21.9M 5s 2024-09-17T23:32:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29550K .......... .......... .......... .......... .......... 17% 27.7M 5s 2024-09-17T23:32:48.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29600K .......... .......... .......... .......... .......... 17% 25.0M 5s 2024-09-17T23:32:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29650K .......... .......... .......... .......... .......... 17% 25.3M 5s 2024-09-17T23:32:48.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29700K .......... .......... .......... .......... .......... 17% 77.2M 5s 2024-09-17T23:32:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29750K .......... .......... .......... .......... .......... 17% 30.1M 5s 2024-09-17T23:32:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29800K .......... .......... .......... .......... .......... 17% 26.1M 5s 2024-09-17T23:32:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29850K .......... .......... .......... .......... .......... 17% 25.3M 5s 2024-09-17T23:32:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29900K .......... .......... .......... .......... .......... 17% 54.4M 5s 2024-09-17T23:32:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 29950K .......... .......... .......... .......... .......... 17% 23.0M 5s 2024-09-17T23:32:48.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30000K .......... .......... .......... .......... .......... 17% 24.2M 5s 2024-09-17T23:32:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30050K .......... .......... .......... .......... .......... 17% 36.6M 5s 2024-09-17T23:32:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30100K .......... .......... .......... .......... .......... 17% 23.3M 5s 2024-09-17T23:32:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30150K .......... .......... .......... .......... .......... 17% 23.9M 5s 2024-09-17T23:32:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30200K .......... .......... .......... .......... .......... 17% 59.2M 5s 2024-09-17T23:32:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30250K .......... .......... .......... .......... .......... 17% 23.9M 5s 2024-09-17T23:32:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30300K .......... .......... .......... .......... .......... 18% 37.2M 5s 2024-09-17T23:32:48.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30350K .......... .......... .......... .......... .......... 18% 22.1M 5s 2024-09-17T23:32:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30400K .......... .......... .......... .......... .......... 18% 26.7M 5s 2024-09-17T23:32:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30450K .......... .......... .......... .......... .......... 18% 53.9M 5s 2024-09-17T23:32:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30500K .......... .......... .......... .......... .......... 18% 23.4M 5s 2024-09-17T23:32:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30550K .......... .......... .......... .......... .......... 18% 40.6M 5s 2024-09-17T23:32:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30600K .......... .......... .......... .......... .......... 18% 21.7M 5s 2024-09-17T23:32:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30650K .......... .......... .......... .......... .......... 18% 24.3M 5s 2024-09-17T23:32:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30700K .......... .......... .......... .......... .......... 18% 55.8M 5s 2024-09-17T23:32:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30750K .......... .......... .......... .......... .......... 18% 23.3M 5s 2024-09-17T23:32:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30800K .......... .......... .......... .......... .......... 18% 23.4M 5s 2024-09-17T23:32:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30850K .......... .......... .......... .......... .......... 18% 37.7M 5s 2024-09-17T23:32:48.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30900K .......... .......... .......... .......... .......... 18% 26.6M 5s 2024-09-17T23:32:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 30950K .......... .......... .......... .......... .......... 18% 40.8M 5s 2024-09-17T23:32:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31000K .......... .......... .......... .......... .......... 18% 25.3M 5s 2024-09-17T23:32:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31050K .......... .......... .......... .......... .......... 18% 40.4M 5s 2024-09-17T23:32:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31100K .......... .......... .......... .......... .......... 18% 23.0M 5s 2024-09-17T23:32:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31150K .......... .......... .......... .......... .......... 18% 22.3M 5s 2024-09-17T23:32:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31200K .......... .......... .......... .......... .......... 18% 24.0M 5s 2024-09-17T23:32:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31250K .......... .......... .......... .......... .......... 18% 64.4M 5s 2024-09-17T23:32:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31300K .......... .......... .......... .......... .......... 18% 21.8M 5s 2024-09-17T23:32:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31350K .......... .......... .......... .......... .......... 18% 35.6M 5s 2024-09-17T23:32:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31400K .......... .......... .......... .......... .......... 18% 25.9M 5s 2024-09-17T23:32:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31450K .......... .......... .......... .......... .......... 18% 51.0M 5s 2024-09-17T23:32:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31500K .......... .......... .......... .......... .......... 18% 21.7M 5s 2024-09-17T23:32:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31550K .......... .......... .......... .......... .......... 18% 26.2M 5s 2024-09-17T23:32:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31600K .......... .......... .......... .......... .......... 18% 40.1M 5s 2024-09-17T23:32:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31650K .......... .......... .......... .......... .......... 18% 23.5M 5s 2024-09-17T23:32:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31700K .......... .......... .......... .......... .......... 18% 51.1M 5s 2024-09-17T23:32:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31750K .......... .......... .......... .......... .......... 18% 23.6M 5s 2024-09-17T23:32:48.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31800K .......... .......... .......... .......... .......... 18% 39.1M 5s 2024-09-17T23:32:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31850K .......... .......... .......... .......... .......... 18% 22.1M 5s 2024-09-17T23:32:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31900K .......... .......... .......... .......... .......... 18% 24.9M 5s 2024-09-17T23:32:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 31950K .......... .......... .......... .......... .......... 18% 23.9M 5s 2024-09-17T23:32:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32000K .......... .......... .......... .......... .......... 19% 54.2M 5s 2024-09-17T23:32:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32050K .......... .......... .......... .......... .......... 19% 24.2M 5s 2024-09-17T23:32:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32100K .......... .......... .......... .......... .......... 19% 40.4M 5s 2024-09-17T23:32:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32150K .......... .......... .......... .......... .......... 19% 21.9M 5s 2024-09-17T23:32:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32200K .......... .......... .......... .......... .......... 19% 63.0M 5s 2024-09-17T23:32:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32250K .......... .......... .......... .......... .......... 19% 22.4M 5s 2024-09-17T23:32:48.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32300K .......... .......... .......... .......... .......... 19% 41.7M 5s 2024-09-17T23:32:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32350K .......... .......... .......... .......... .......... 19% 21.5M 5s 2024-09-17T23:32:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32400K .......... .......... .......... .......... .......... 19% 23.6M 5s 2024-09-17T23:32:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32450K .......... .......... .......... .......... .......... 19% 27.2M 5s 2024-09-17T23:32:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32500K .......... .......... .......... .......... .......... 19% 43.0M 5s 2024-09-17T23:32:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32550K .......... .......... .......... .......... .......... 19% 26.0M 5s 2024-09-17T23:32:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32600K .......... .......... .......... .......... .......... 19% 35.7M 5s 2024-09-17T23:32:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32650K .......... .......... .......... .......... .......... 19% 24.2M 5s 2024-09-17T23:32:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32700K .......... .......... .......... .......... .......... 19% 55.0M 5s 2024-09-17T23:32:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32750K .......... .......... .......... .......... .......... 19% 24.3M 5s 2024-09-17T23:32:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32800K .......... .......... .......... .......... .......... 19% 23.1M 5s 2024-09-17T23:32:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32850K .......... .......... .......... .......... .......... 19% 26.1M 5s 2024-09-17T23:32:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32900K .......... .......... .......... .......... .......... 19% 32.4M 5s 2024-09-17T23:32:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 32950K .......... .......... .......... .......... .......... 19% 23.9M 5s 2024-09-17T23:32:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33000K .......... .......... .......... .......... .......... 19% 75.9M 5s 2024-09-17T23:32:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33050K .......... .......... .......... .......... .......... 19% 20.8M 5s 2024-09-17T23:32:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33100K .......... .......... .......... .......... .......... 19% 37.3M 5s 2024-09-17T23:32:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33150K .......... .......... .......... .......... .......... 19% 23.7M 5s 2024-09-17T23:32:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33200K .......... .......... .......... .......... .......... 19% 25.5M 5s 2024-09-17T23:32:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33250K .......... .......... .......... .......... .......... 19% 48.3M 5s 2024-09-17T23:32:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33300K .......... .......... .......... .......... .......... 19% 27.2M 5s 2024-09-17T23:32:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33350K .......... .......... .......... .......... .......... 19% 30.0M 5s 2024-09-17T23:32:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33400K .......... .......... .......... .......... .......... 19% 24.9M 5s 2024-09-17T23:32:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33450K .......... .......... .......... .......... .......... 19% 22.4M 5s 2024-09-17T23:32:48.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33500K .......... .......... .......... .......... .......... 19% 62.9M 5s 2024-09-17T23:32:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33550K .......... .......... .......... .......... .......... 19% 23.5M 5s 2024-09-17T23:32:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33600K .......... .......... .......... .......... .......... 19% 23.6M 5s 2024-09-17T23:32:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33650K .......... .......... .......... .......... .......... 19% 39.5M 5s 2024-09-17T23:32:48.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33700K .......... .......... .......... .......... .......... 20% 23.6M 5s 2024-09-17T23:32:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33750K .......... .......... .......... .......... .......... 20% 48.1M 5s 2024-09-17T23:32:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33800K .......... .......... .......... .......... .......... 20% 24.6M 5s 2024-09-17T23:32:48.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33850K .......... .......... .......... .......... .......... 20% 39.6M 5s 2024-09-17T23:32:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33900K .......... .......... .......... .......... .......... 20% 23.2M 5s 2024-09-17T23:32:48.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 33950K .......... .......... .......... .......... .......... 20% 24.3M 5s 2024-09-17T23:32:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34000K .......... .......... .......... .......... .......... 20% 22.3M 5s 2024-09-17T23:32:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34050K .......... .......... .......... .......... .......... 20% 53.8M 5s 2024-09-17T23:32:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34100K .......... .......... .......... .......... .......... 20% 36.5M 5s 2024-09-17T23:32:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34150K .......... .......... .......... .......... .......... 20% 25.2M 5s 2024-09-17T23:32:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34200K .......... .......... .......... .......... .......... 20% 23.5M 5s 2024-09-17T23:32:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34250K .......... .......... .......... .......... .......... 20% 58.1M 5s 2024-09-17T23:32:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34300K .......... .......... .......... .......... .......... 20% 23.9M 5s 2024-09-17T23:32:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34350K .......... .......... .......... .......... .......... 20% 23.3M 5s 2024-09-17T23:32:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34400K .......... .......... .......... .......... .......... 20% 38.1M 5s 2024-09-17T23:32:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34450K .......... .......... .......... .......... .......... 20% 23.6M 5s 2024-09-17T23:32:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34500K .......... .......... .......... .......... .......... 20% 56.5M 5s 2024-09-17T23:32:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34550K .......... .......... .......... .......... .......... 20% 23.5M 5s 2024-09-17T23:32:48.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34600K .......... .......... .......... .......... .......... 20% 37.0M 5s 2024-09-17T23:32:48.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34650K .......... .......... .......... .......... .......... 20% 23.3M 5s 2024-09-17T23:32:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34700K .......... .......... .......... .......... .......... 20% 25.3M 5s 2024-09-17T23:32:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34750K .......... .......... .......... .......... .......... 20% 23.3M 5s 2024-09-17T23:32:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34800K .......... .......... .......... .......... .......... 20% 47.9M 5s 2024-09-17T23:32:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34850K .......... .......... .......... .......... .......... 20% 25.2M 5s 2024-09-17T23:32:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34900K .......... .......... .......... .......... .......... 20% 36.3M 5s 2024-09-17T23:32:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 34950K .......... .......... .......... .......... .......... 20% 24.4M 5s 2024-09-17T23:32:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35000K .......... .......... .......... .......... .......... 20% 47.5M 5s 2024-09-17T23:32:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35050K .......... .......... .......... .......... .......... 20% 22.9M 5s 2024-09-17T23:32:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35100K .......... .......... .......... .......... .......... 20% 36.5M 5s 2024-09-17T23:32:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35150K .......... .......... .......... .......... .......... 20% 24.9M 5s 2024-09-17T23:32:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35200K .......... .......... .......... .......... .......... 20% 25.1M 5s 2024-09-17T23:32:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35250K .......... .......... .......... .......... .......... 20% 24.0M 5s 2024-09-17T23:32:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35300K .......... .......... .......... .......... .......... 20% 47.5M 5s 2024-09-17T23:32:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35350K .......... .......... .......... .......... .......... 21% 24.2M 5s 2024-09-17T23:32:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35400K .......... .......... .......... .......... .......... 21% 37.6M 5s 2024-09-17T23:32:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35450K .......... .......... .......... .......... .......... 21% 24.5M 5s 2024-09-17T23:32:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35500K .......... .......... .......... .......... .......... 21% 51.7M 5s 2024-09-17T23:32:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35550K .......... .......... .......... .......... .......... 21% 23.0M 5s 2024-09-17T23:32:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35600K .......... .......... .......... .......... .......... 21% 23.6M 5s 2024-09-17T23:32:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35650K .......... .......... .......... .......... .......... 21% 39.3M 5s 2024-09-17T23:32:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35700K .......... .......... .......... .......... .......... 21% 23.8M 5s 2024-09-17T23:32:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35750K .......... .......... .......... .......... .......... 21% 30.8M 5s 2024-09-17T23:32:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35800K .......... .......... .......... .......... .......... 21% 36.9M 5s 2024-09-17T23:32:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35850K .......... .......... .......... .......... .......... 21% 22.5M 5s 2024-09-17T23:32:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35900K .......... .......... .......... .......... .......... 21% 38.5M 5s 2024-09-17T23:32:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 35950K .......... .......... .......... .......... .......... 21% 25.3M 5s 2024-09-17T23:32:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36000K .......... .......... .......... .......... .......... 21% 26.4M 5s 2024-09-17T23:32:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36050K .......... .......... .......... .......... .......... 21% 36.1M 5s 2024-09-17T23:32:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36100K .......... .......... .......... .......... .......... 21% 24.9M 5s 2024-09-17T23:32:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36150K .......... .......... .......... .......... .......... 21% 37.6M 5s 2024-09-17T23:32:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36200K .......... .......... .......... .......... .......... 21% 20.1M 5s 2024-09-17T23:32:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36250K .......... .......... .......... .......... .......... 21% 85.5M 5s 2024-09-17T23:32:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36300K .......... .......... .......... .......... .......... 21% 24.3M 5s 2024-09-17T23:32:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36350K .......... .......... .......... .......... .......... 21% 22.5M 5s 2024-09-17T23:32:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36400K .......... .......... .......... .......... .......... 21% 24.4M 5s 2024-09-17T23:32:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36450K .......... .......... .......... .......... .......... 21% 38.5M 5s 2024-09-17T23:32:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36500K .......... .......... .......... .......... .......... 21% 27.0M 5s 2024-09-17T23:32:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36550K .......... .......... .......... .......... .......... 21% 41.8M 5s 2024-09-17T23:32:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36600K .......... .......... .......... .......... .......... 21% 23.2M 5s 2024-09-17T23:32:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36650K .......... .......... .......... .......... .......... 21% 35.3M 5s 2024-09-17T23:32:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36700K .......... .......... .......... .......... .......... 21% 28.8M 5s 2024-09-17T23:32:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36750K .......... .......... .......... .......... .......... 21% 21.0M 5s 2024-09-17T23:32:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36800K .......... .......... .......... .......... .......... 21% 59.4M 5s 2024-09-17T23:32:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36850K .......... .......... .......... .......... .......... 21% 22.2M 5s 2024-09-17T23:32:48.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36900K .......... .......... .......... .......... .......... 21% 43.5M 5s 2024-09-17T23:32:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 36950K .......... .......... .......... .......... .......... 21% 23.4M 5s 2024-09-17T23:32:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37000K .......... .......... .......... .......... .......... 21% 38.7M 5s 2024-09-17T23:32:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37050K .......... .......... .......... .......... .......... 22% 29.7M 5s 2024-09-17T23:32:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37100K .......... .......... .......... .......... .......... 22% 29.0M 5s 2024-09-17T23:32:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37150K .......... .......... .......... .......... .......... 22% 25.1M 5s 2024-09-17T23:32:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37200K .......... .......... .......... .......... .......... 22% 23.6M 5s 2024-09-17T23:32:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37250K .......... .......... .......... .......... .......... 22% 34.4M 5s 2024-09-17T23:32:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37300K .......... .......... .......... .......... .......... 22% 40.9M 5s 2024-09-17T23:32:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37350K .......... .......... .......... .......... .......... 22% 25.3M 5s 2024-09-17T23:32:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37400K .......... .......... .......... .......... .......... 22% 25.2M 5s 2024-09-17T23:32:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37450K .......... .......... .......... .......... .......... 22% 23.1M 5s 2024-09-17T23:32:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37500K .......... .......... .......... .......... .......... 22% 51.9M 5s 2024-09-17T23:32:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37550K .......... .......... .......... .......... .......... 22% 17.9M 5s 2024-09-17T23:32:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37600K .......... .......... .......... .......... .......... 22% 54.7M 5s 2024-09-17T23:32:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37650K .......... .......... .......... .......... .......... 22% 25.4M 5s 2024-09-17T23:32:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37700K .......... .......... .......... .......... .......... 22% 26.4M 5s 2024-09-17T23:32:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37750K .......... .......... .......... .......... .......... 22% 41.4M 5s 2024-09-17T23:32:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37800K .......... .......... .......... .......... .......... 22% 38.0M 5s 2024-09-17T23:32:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37850K .......... .......... .......... .......... .......... 22% 22.5M 5s 2024-09-17T23:32:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37900K .......... .......... .......... .......... .......... 22% 40.5M 5s 2024-09-17T23:32:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 37950K .......... .......... .......... .......... .......... 22% 22.4M 5s 2024-09-17T23:32:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38000K .......... .......... .......... .......... .......... 22% 24.6M 5s 2024-09-17T23:32:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38050K .......... .......... .......... .......... .......... 22% 36.8M 5s 2024-09-17T23:32:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38100K .......... .......... .......... .......... .......... 22% 30.1M 5s 2024-09-17T23:32:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38150K .......... .......... .......... .......... .......... 22% 21.3M 5s 2024-09-17T23:32:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38200K .......... .......... .......... .......... .......... 22% 45.2M 5s 2024-09-17T23:32:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38250K .......... .......... .......... .......... .......... 22% 22.8M 5s 2024-09-17T23:32:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38300K .......... .......... .......... .......... .......... 22% 55.3M 5s 2024-09-17T23:32:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38350K .......... .......... .......... .......... .......... 22% 23.5M 5s 2024-09-17T23:32:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38400K .......... .......... .......... .......... .......... 22% 33.7M 5s 2024-09-17T23:32:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38450K .......... .......... .......... .......... .......... 22% 25.7M 5s 2024-09-17T23:32:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38500K .......... .......... .......... .......... .......... 22% 24.2M 5s 2024-09-17T23:32:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38550K .......... .......... .......... .......... .......... 22% 23.4M 5s 2024-09-17T23:32:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38600K .......... .......... .......... .......... .......... 22% 53.6M 5s 2024-09-17T23:32:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38650K .......... .......... .......... .......... .......... 22% 24.1M 5s 2024-09-17T23:32:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38700K .......... .......... .......... .......... .......... 22% 39.2M 5s 2024-09-17T23:32:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38750K .......... .......... .......... .......... .......... 23% 22.9M 5s 2024-09-17T23:32:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38800K .......... .......... .......... .......... .......... 23% 24.7M 5s 2024-09-17T23:32:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38850K .......... .......... .......... .......... .......... 23% 46.5M 5s 2024-09-17T23:32:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38900K .......... .......... .......... .......... .......... 23% 25.0M 5s 2024-09-17T23:32:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 38950K .......... .......... .......... .......... .......... 23% 35.1M 5s 2024-09-17T23:32:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39000K .......... .......... .......... .......... .......... 23% 24.4M 5s 2024-09-17T23:32:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39050K .......... .......... .......... .......... .......... 23% 54.3M 5s 2024-09-17T23:32:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39100K .......... .......... .......... .......... .......... 23% 22.9M 5s 2024-09-17T23:32:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39150K .......... .......... .......... .......... .......... 23% 25.1M 5s 2024-09-17T23:32:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39200K .......... .......... .......... .......... .......... 23% 38.6M 5s 2024-09-17T23:32:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39250K .......... .......... .......... .......... .......... 23% 24.1M 5s 2024-09-17T23:32:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39300K .......... .......... .......... .......... .......... 23% 21.5M 5s 2024-09-17T23:32:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39350K .......... .......... .......... .......... .......... 23% 54.6M 5s 2024-09-17T23:32:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39400K .......... .......... .......... .......... .......... 23% 24.7M 5s 2024-09-17T23:32:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39450K .......... .......... .......... .......... .......... 23% 40.8M 5s 2024-09-17T23:32:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39500K .......... .......... .......... .......... .......... 23% 23.7M 5s 2024-09-17T23:32:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39550K .......... .......... .......... .......... .......... 23% 22.8M 5s 2024-09-17T23:32:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39600K .......... .......... .......... .......... .......... 23% 53.7M 5s 2024-09-17T23:32:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39650K .......... .......... .......... .......... .......... 23% 24.3M 5s 2024-09-17T23:32:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39700K .......... .......... .......... .......... .......... 23% 35.9M 5s 2024-09-17T23:32:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39750K .......... .......... .......... .......... .......... 23% 23.3M 5s 2024-09-17T23:32:49.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39800K .......... .......... .......... .......... .......... 23% 23.7M 5s 2024-09-17T23:32:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39850K .......... .......... .......... .......... .......... 23% 57.7M 5s 2024-09-17T23:32:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39900K .......... .......... .......... .......... .......... 23% 23.8M 5s 2024-09-17T23:32:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 39950K .......... .......... .......... .......... .......... 23% 24.4M 5s 2024-09-17T23:32:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40000K .......... .......... .......... .......... .......... 23% 37.7M 5s 2024-09-17T23:32:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40050K .......... .......... .......... .......... .......... 23% 23.7M 5s 2024-09-17T23:32:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40100K .......... .......... .......... .......... .......... 23% 51.3M 5s 2024-09-17T23:32:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40150K .......... .......... .......... .......... .......... 23% 23.9M 5s 2024-09-17T23:32:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40200K .......... .......... .......... .......... .......... 23% 39.5M 5s 2024-09-17T23:32:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40250K .......... .......... .......... .......... .......... 23% 22.9M 5s 2024-09-17T23:32:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40300K .......... .......... .......... .......... .......... 23% 23.3M 5s 2024-09-17T23:32:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40350K .......... .......... .......... .......... .......... 23% 25.4M 5s 2024-09-17T23:32:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40400K .......... .......... .......... .......... .......... 23% 52.1M 5s 2024-09-17T23:32:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40450K .......... .......... .......... .......... .......... 24% 23.7M 5s 2024-09-17T23:32:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40500K .......... .......... .......... .......... .......... 24% 38.3M 5s 2024-09-17T23:32:49.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40550K .......... .......... .......... .......... .......... 24% 23.1M 5s 2024-09-17T23:32:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40600K .......... .......... .......... .......... .......... 24% 54.7M 5s 2024-09-17T23:32:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40650K .......... .......... .......... .......... .......... 24% 22.8M 5s 2024-09-17T23:32:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40700K .......... .......... .......... .......... .......... 24% 38.1M 5s 2024-09-17T23:32:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40750K .......... .......... .......... .......... .......... 24% 23.9M 5s 2024-09-17T23:32:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40800K .......... .......... .......... .......... .......... 24% 23.2M 4s 2024-09-17T23:32:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40850K .......... .......... .......... .......... .......... 24% 25.1M 4s 2024-09-17T23:32:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40900K .......... .......... .......... .......... .......... 24% 51.6M 4s 2024-09-17T23:32:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 40950K .......... .......... .......... .......... .......... 24% 23.6M 4s 2024-09-17T23:32:49.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41000K .......... .......... .......... .......... .......... 24% 37.0M 4s 2024-09-17T23:32:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41050K .......... .......... .......... .......... .......... 24% 23.9M 4s 2024-09-17T23:32:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41100K .......... .......... .......... .......... .......... 24% 47.3M 4s 2024-09-17T23:32:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41150K .......... .......... .......... .......... .......... 24% 26.2M 4s 2024-09-17T23:32:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41200K .......... .......... .......... .......... .......... 24% 23.7M 4s 2024-09-17T23:32:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41250K .......... .......... .......... .......... .......... 24% 34.6M 4s 2024-09-17T23:32:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41300K .......... .......... .......... .......... .......... 24% 26.0M 4s 2024-09-17T23:32:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41350K .......... .......... .......... .......... .......... 24% 23.1M 4s 2024-09-17T23:32:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41400K .......... .......... .......... .......... .......... 24% 50.5M 4s 2024-09-17T23:32:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41450K .......... .......... .......... .......... .......... 24% 36.2M 4s 2024-09-17T23:32:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41500K .......... .......... .......... .......... .......... 24% 23.9M 4s 2024-09-17T23:32:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41550K .......... .......... .......... .......... .......... 24% 23.4M 4s 2024-09-17T23:32:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41600K .......... .......... .......... .......... .......... 24% 25.8M 4s 2024-09-17T23:32:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41650K .......... .......... .......... .......... .......... 24% 50.7M 4s 2024-09-17T23:32:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41700K .......... .......... .......... .......... .......... 24% 23.9M 4s 2024-09-17T23:32:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41750K .......... .......... .......... .......... .......... 24% 33.5M 4s 2024-09-17T23:32:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41800K .......... .......... .......... .......... .......... 24% 24.9M 4s 2024-09-17T23:32:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41850K .......... .......... .......... .......... .......... 24% 53.5M 4s 2024-09-17T23:32:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41900K .......... .......... .......... .......... .......... 24% 25.3M 4s 2024-09-17T23:32:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 41950K .......... .......... .......... .......... .......... 24% 23.3M 4s 2024-09-17T23:32:49.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42000K .......... .......... .......... .......... .......... 24% 37.5M 4s 2024-09-17T23:32:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42050K .......... .......... .......... .......... .......... 24% 22.8M 4s 2024-09-17T23:32:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42100K .......... .......... .......... .......... .......... 25% 25.1M 4s 2024-09-17T23:32:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42150K .......... .......... .......... .......... .......... 25% 46.0M 4s 2024-09-17T23:32:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42200K .......... .......... .......... .......... .......... 25% 24.5M 4s 2024-09-17T23:32:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42250K .......... .......... .......... .......... .......... 25% 39.8M 4s 2024-09-17T23:32:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42300K .......... .......... .......... .......... .......... 25% 24.8M 4s 2024-09-17T23:32:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42350K .......... .......... .......... .......... .......... 25% 23.0M 4s 2024-09-17T23:32:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42400K .......... .......... .......... .......... .......... 25% 52.7M 4s 2024-09-17T23:32:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42450K .......... .......... .......... .......... .......... 25% 22.9M 4s 2024-09-17T23:32:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42500K .......... .......... .......... .......... .......... 25% 37.0M 4s 2024-09-17T23:32:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42550K .......... .......... .......... .......... .......... 25% 23.2M 4s 2024-09-17T23:32:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42600K .......... .......... .......... .......... .......... 25% 25.3M 4s 2024-09-17T23:32:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42650K .......... .......... .......... .......... .......... 25% 56.0M 4s 2024-09-17T23:32:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42700K .......... .......... .......... .......... .......... 25% 23.2M 4s 2024-09-17T23:32:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42750K .......... .......... .......... .......... .......... 25% 24.6M 4s 2024-09-17T23:32:49.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42800K .......... .......... .......... .......... .......... 25% 40.6M 4s 2024-09-17T23:32:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42850K .......... .......... .......... .......... .......... 25% 22.7M 4s 2024-09-17T23:32:49.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42900K .......... .......... .......... .......... .......... 25% 50.8M 4s 2024-09-17T23:32:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 42950K .......... .......... .......... .......... .......... 25% 24.3M 4s 2024-09-17T23:32:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43000K .......... .......... .......... .......... .......... 25% 40.3M 4s 2024-09-17T23:32:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43050K .......... .......... .......... .......... .......... 25% 22.0M 4s 2024-09-17T23:32:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43100K .......... .......... .......... .......... .......... 25% 24.2M 4s 2024-09-17T23:32:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43150K .......... .......... .......... .......... .......... 25% 24.3M 4s 2024-09-17T23:32:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43200K .......... .......... .......... .......... .......... 25% 53.4M 4s 2024-09-17T23:32:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43250K .......... .......... .......... .......... .......... 25% 25.7M 4s 2024-09-17T23:32:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43300K .......... .......... .......... .......... .......... 25% 33.5M 4s 2024-09-17T23:32:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43350K .......... .......... .......... .......... .......... 25% 23.2M 4s 2024-09-17T23:32:49.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43400K .......... .......... .......... .......... .......... 25% 51.1M 4s 2024-09-17T23:32:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43450K .......... .......... .......... .......... .......... 25% 24.8M 4s 2024-09-17T23:32:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43500K .......... .......... .......... .......... .......... 25% 37.3M 4s 2024-09-17T23:32:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43550K .......... .......... .......... .......... .......... 25% 24.0M 4s 2024-09-17T23:32:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43600K .......... .......... .......... .......... .......... 25% 25.7M 4s 2024-09-17T23:32:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43650K .......... .......... .......... .......... .......... 25% 21.7M 4s 2024-09-17T23:32:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43700K .......... .......... .......... .......... .......... 25% 55.4M 4s 2024-09-17T23:32:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43750K .......... .......... .......... .......... .......... 25% 38.6M 4s 2024-09-17T23:32:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43800K .......... .......... .......... .......... .......... 26% 23.6M 4s 2024-09-17T23:32:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43850K .......... .......... .......... .......... .......... 26% 23.7M 4s 2024-09-17T23:32:49.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43900K .......... .......... .......... .......... .......... 26% 55.6M 4s 2024-09-17T23:32:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 43950K .......... .......... .......... .......... .......... 26% 23.7M 4s 2024-09-17T23:32:49.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44000K .......... .......... .......... .......... .......... 26% 23.8M 4s 2024-09-17T23:32:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44050K .......... .......... .......... .......... .......... 26% 39.0M 4s 2024-09-17T23:32:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44100K .......... .......... .......... .......... .......... 26% 24.7M 4s 2024-09-17T23:32:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44150K .......... .......... .......... .......... .......... 26% 47.6M 4s 2024-09-17T23:32:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44200K .......... .......... .......... .......... .......... 26% 24.0M 4s 2024-09-17T23:32:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44250K .......... .......... .......... .......... .......... 26% 33.1M 4s 2024-09-17T23:32:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44300K .......... .......... .......... .......... .......... 26% 25.7M 4s 2024-09-17T23:32:49.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44350K .......... .......... .......... .......... .......... 26% 22.7M 4s 2024-09-17T23:32:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44400K .......... .......... .......... .......... .......... 26% 26.0M 4s 2024-09-17T23:32:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44450K .......... .......... .......... .......... .......... 26% 50.8M 4s 2024-09-17T23:32:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44500K .......... .......... .......... .......... .......... 26% 23.8M 4s 2024-09-17T23:32:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44550K .......... .......... .......... .......... .......... 26% 38.4M 4s 2024-09-17T23:32:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44600K .......... .......... .......... .......... .......... 26% 23.2M 4s 2024-09-17T23:32:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44650K .......... .......... .......... .......... .......... 26% 56.8M 4s 2024-09-17T23:32:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44700K .......... .......... .......... .......... .......... 26% 22.6M 4s 2024-09-17T23:32:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44750K .......... .......... .......... .......... .......... 26% 23.8M 4s 2024-09-17T23:32:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44800K .......... .......... .......... .......... .......... 26% 37.9M 4s 2024-09-17T23:32:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44850K .......... .......... .......... .......... .......... 26% 23.3M 4s 2024-09-17T23:32:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44900K .......... .......... .......... .......... .......... 26% 24.5M 4s 2024-09-17T23:32:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 44950K .......... .......... .......... .......... .......... 26% 56.3M 4s 2024-09-17T23:32:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45000K .......... .......... .......... .......... .......... 26% 22.9M 4s 2024-09-17T23:32:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45050K .......... .......... .......... .......... .......... 26% 39.5M 4s 2024-09-17T23:32:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45100K .......... .......... .......... .......... .......... 26% 24.4M 4s 2024-09-17T23:32:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45150K .......... .......... .......... .......... .......... 26% 22.7M 4s 2024-09-17T23:32:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45200K .......... .......... .......... .......... .......... 26% 50.1M 4s 2024-09-17T23:32:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45250K .......... .......... .......... .......... .......... 26% 24.5M 4s 2024-09-17T23:32:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45300K .......... .......... .......... .......... .......... 26% 37.3M 4s 2024-09-17T23:32:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45350K .......... .......... .......... .......... .......... 26% 22.7M 4s 2024-09-17T23:32:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45400K .......... .......... .......... .......... .......... 26% 24.0M 4s 2024-09-17T23:32:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45450K .......... .......... .......... .......... .......... 26% 54.5M 4s 2024-09-17T23:32:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45500K .......... .......... .......... .......... .......... 27% 22.7M 4s 2024-09-17T23:32:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45550K .......... .......... .......... .......... .......... 27% 25.1M 4s 2024-09-17T23:32:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45600K .......... .......... .......... .......... .......... 27% 34.2M 4s 2024-09-17T23:32:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45650K .......... .......... .......... .......... .......... 27% 25.9M 4s 2024-09-17T23:32:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45700K .......... .......... .......... .......... .......... 27% 49.3M 4s 2024-09-17T23:32:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45750K .......... .......... .......... .......... .......... 27% 23.7M 4s 2024-09-17T23:32:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45800K .......... .......... .......... .......... .......... 27% 23.8M 4s 2024-09-17T23:32:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45850K .......... .......... .......... .......... .......... 27% 28.5M 4s 2024-09-17T23:32:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45900K .......... .......... .......... .......... .......... 27% 113M 4s 2024-09-17T23:32:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 45950K .......... .......... .......... .......... .......... 27% 15.0M 4s 2024-09-17T23:32:49.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46000K .......... .......... .......... .......... .......... 27% 52.5M 4s 2024-09-17T23:32:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46050K .......... .......... .......... .......... .......... 27% 24.7M 4s 2024-09-17T23:32:49.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46100K .......... .......... .......... .......... .......... 27% 32.6M 4s 2024-09-17T23:32:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46150K .......... .......... .......... .......... .......... 27% 25.1M 4s 2024-09-17T23:32:49.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46200K .......... .......... .......... .......... .......... 27% 49.7M 4s 2024-09-17T23:32:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46250K .......... .......... .......... .......... .......... 27% 22.9M 4s 2024-09-17T23:32:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46300K .......... .......... .......... .......... .......... 27% 45.2M 4s 2024-09-17T23:32:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46350K .......... .......... .......... .......... .......... 27% 22.0M 4s 2024-09-17T23:32:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46400K .......... .......... .......... .......... .......... 27% 28.7M 4s 2024-09-17T23:32:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46450K .......... .......... .......... .......... .......... 27% 20.7M 4s 2024-09-17T23:32:49.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46500K .......... .......... .......... .......... .......... 27% 55.2M 4s 2024-09-17T23:32:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46550K .......... .......... .......... .......... .......... 27% 31.6M 4s 2024-09-17T23:32:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46600K .......... .......... .......... .......... .......... 27% 27.1M 4s 2024-09-17T23:32:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46650K .......... .......... .......... .......... .......... 27% 23.0M 4s 2024-09-17T23:32:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46700K .......... .......... .......... .......... .......... 27% 50.1M 4s 2024-09-17T23:32:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46750K .......... .......... .......... .......... .......... 27% 25.2M 4s 2024-09-17T23:32:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46800K .......... .......... .......... .......... .......... 27% 23.4M 4s 2024-09-17T23:32:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46850K .......... .......... .......... .......... .......... 27% 38.5M 4s 2024-09-17T23:32:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46900K .......... .......... .......... .......... .......... 27% 24.8M 4s 2024-09-17T23:32:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 46950K .......... .......... .......... .......... .......... 27% 53.2M 4s 2024-09-17T23:32:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47000K .......... .......... .......... .......... .......... 27% 22.6M 4s 2024-09-17T23:32:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47050K .......... .......... .......... .......... .......... 27% 38.2M 4s 2024-09-17T23:32:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47100K .......... .......... .......... .......... .......... 27% 24.7M 4s 2024-09-17T23:32:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47150K .......... .......... .......... .......... .......... 28% 23.2M 4s 2024-09-17T23:32:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47200K .......... .......... .......... .......... .......... 28% 23.4M 4s 2024-09-17T23:32:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47250K .......... .......... .......... .......... .......... 28% 55.5M 4s 2024-09-17T23:32:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47300K .......... .......... .......... .......... .......... 28% 23.8M 4s 2024-09-17T23:32:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47350K .......... .......... .......... .......... .......... 28% 41.7M 4s 2024-09-17T23:32:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47400K .......... .......... .......... .......... .......... 28% 25.6M 4s 2024-09-17T23:32:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47450K .......... .......... .......... .......... .......... 28% 35.0M 4s 2024-09-17T23:32:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47500K .......... .......... .......... .......... .......... 28% 26.0M 4s 2024-09-17T23:32:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47550K .......... .......... .......... .......... .......... 28% 24.2M 4s 2024-09-17T23:32:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47600K .......... .......... .......... .......... .......... 28% 36.2M 4s 2024-09-17T23:32:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47650K .......... .......... .......... .......... .......... 28% 22.9M 4s 2024-09-17T23:32:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47700K .......... .......... .......... .......... .......... 28% 24.9M 4s 2024-09-17T23:32:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47750K .......... .......... .......... .......... .......... 28% 51.2M 4s 2024-09-17T23:32:49.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47800K .......... .......... .......... .......... .......... 28% 24.8M 4s 2024-09-17T23:32:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47850K .......... .......... .......... .......... .......... 28% 38.9M 4s 2024-09-17T23:32:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47900K .......... .......... .......... .......... .......... 28% 23.4M 4s 2024-09-17T23:32:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 47950K .......... .......... .......... .......... .......... 28% 23.7M 4s 2024-09-17T23:32:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48000K .......... .......... .......... .......... .......... 28% 46.5M 4s 2024-09-17T23:32:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48050K .......... .......... .......... .......... .......... 28% 24.6M 4s 2024-09-17T23:32:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48100K .......... .......... .......... .......... .......... 28% 39.1M 4s 2024-09-17T23:32:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48150K .......... .......... .......... .......... .......... 28% 20.5M 4s 2024-09-17T23:32:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48200K .......... .......... .......... .......... .......... 28% 29.0M 4s 2024-09-17T23:32:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48250K .......... .......... .......... .......... .......... 28% 52.6M 4s 2024-09-17T23:32:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48300K .......... .......... .......... .......... .......... 28% 24.3M 4s 2024-09-17T23:32:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48350K .......... .......... .......... .......... .......... 28% 22.6M 4s 2024-09-17T23:32:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48400K .......... .......... .......... .......... .......... 28% 38.3M 4s 2024-09-17T23:32:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48450K .......... .......... .......... .......... .......... 28% 25.6M 4s 2024-09-17T23:32:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48500K .......... .......... .......... .......... .......... 28% 43.0M 4s 2024-09-17T23:32:49.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48550K .......... .......... .......... .......... .......... 28% 24.4M 4s 2024-09-17T23:32:49.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48600K .......... .......... .......... .......... .......... 28% 42.5M 4s 2024-09-17T23:32:49.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48650K .......... .......... .......... .......... .......... 28% 24.3M 4s 2024-09-17T23:32:49.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48700K .......... .......... .......... .......... .......... 28% 48.9M 4s 2024-09-17T23:32:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48750K .......... .......... .......... .......... .......... 28% 15.1M 4s 2024-09-17T23:32:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48800K .......... .......... .......... .......... .......... 28% 51.2M 4s 2024-09-17T23:32:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48850K .......... .......... .......... .......... .......... 29% 24.1M 4s 2024-09-17T23:32:49.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48900K .......... .......... .......... .......... .......... 29% 41.6M 4s 2024-09-17T23:32:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 48950K .......... .......... .......... .......... .......... 29% 23.6M 4s 2024-09-17T23:32:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49000K .......... .......... .......... .......... .......... 29% 49.5M 4s 2024-09-17T23:32:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49050K .......... .......... .......... .......... .......... 29% 22.6M 4s 2024-09-17T23:32:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49100K .......... .......... .......... .......... .......... 29% 43.3M 4s 2024-09-17T23:32:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49150K .......... .......... .......... .......... .......... 29% 23.3M 4s 2024-09-17T23:32:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49200K .......... .......... .......... .......... .......... 29% 24.1M 4s 2024-09-17T23:32:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49250K .......... .......... .......... .......... .......... 29% 45.2M 4s 2024-09-17T23:32:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49300K .......... .......... .......... .......... .......... 29% 25.2M 4s 2024-09-17T23:32:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49350K .......... .......... .......... .......... .......... 29% 38.8M 4s 2024-09-17T23:32:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49400K .......... .......... .......... .......... .......... 29% 23.4M 4s 2024-09-17T23:32:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49450K .......... .......... .......... .......... .......... 29% 24.5M 4s 2024-09-17T23:32:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49500K .......... .......... .......... .......... .......... 29% 52.4M 4s 2024-09-17T23:32:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49550K .......... .......... .......... .......... .......... 29% 23.4M 4s 2024-09-17T23:32:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49600K .......... .......... .......... .......... .......... 29% 24.3M 4s 2024-09-17T23:32:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49650K .......... .......... .......... .......... .......... 29% 34.0M 4s 2024-09-17T23:32:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49700K .......... .......... .......... .......... .......... 29% 25.1M 4s 2024-09-17T23:32:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49750K .......... .......... .......... .......... .......... 29% 55.2M 4s 2024-09-17T23:32:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49800K .......... .......... .......... .......... .......... 29% 22.8M 4s 2024-09-17T23:32:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49850K .......... .......... .......... .......... .......... 29% 38.1M 4s 2024-09-17T23:32:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49900K .......... .......... .......... .......... .......... 29% 25.5M 4s 2024-09-17T23:32:49.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 49950K .......... .......... .......... .......... .......... 29% 21.7M 4s 2024-09-17T23:32:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50000K .......... .......... .......... .......... .......... 29% 26.4M 4s 2024-09-17T23:32:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50050K .......... .......... .......... .......... .......... 29% 52.3M 4s 2024-09-17T23:32:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50100K .......... .......... .......... .......... .......... 29% 22.1M 4s 2024-09-17T23:32:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50150K .......... .......... .......... .......... .......... 29% 42.6M 4s 2024-09-17T23:32:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50200K .......... .......... .......... .......... .......... 29% 23.4M 4s 2024-09-17T23:32:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50250K .......... .......... .......... .......... .......... 29% 53.5M 4s 2024-09-17T23:32:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50300K .......... .......... .......... .......... .......... 29% 23.1M 4s 2024-09-17T23:32:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50350K .......... .......... .......... .......... .......... 29% 24.1M 4s 2024-09-17T23:32:49.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50400K .......... .......... .......... .......... .......... 29% 39.2M 4s 2024-09-17T23:32:49.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50450K .......... .......... .......... .......... .......... 29% 23.3M 4s 2024-09-17T23:32:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50500K .......... .......... .......... .......... .......... 29% 23.1M 4s 2024-09-17T23:32:49.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50550K .......... .......... .......... .......... .......... 30% 57.1M 4s 2024-09-17T23:32:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50600K .......... .......... .......... .......... .......... 30% 23.2M 4s 2024-09-17T23:32:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50650K .......... .......... .......... .......... .......... 30% 37.6M 4s 2024-09-17T23:32:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50700K .......... .......... .......... .......... .......... 30% 24.8M 4s 2024-09-17T23:32:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50750K .......... .......... .......... .......... .......... 30% 24.3M 4s 2024-09-17T23:32:49.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50800K .......... .......... .......... .......... .......... 30% 48.6M 4s 2024-09-17T23:32:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50850K .......... .......... .......... .......... .......... 30% 24.3M 4s 2024-09-17T23:32:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50900K .......... .......... .......... .......... .......... 30% 39.4M 4s 2024-09-17T23:32:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 50950K .......... .......... .......... .......... .......... 30% 22.9M 4s 2024-09-17T23:32:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51000K .......... .......... .......... .......... .......... 30% 26.3M 4s 2024-09-17T23:32:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51050K .......... .......... .......... .......... .......... 30% 42.5M 4s 2024-09-17T23:32:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51100K .......... .......... .......... .......... .......... 30% 23.6M 4s 2024-09-17T23:32:49.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51150K .......... .......... .......... .......... .......... 30% 24.0M 4s 2024-09-17T23:32:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51200K .......... .......... .......... .......... .......... 30% 40.0M 4s 2024-09-17T23:32:49.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51250K .......... .......... .......... .......... .......... 30% 24.0M 4s 2024-09-17T23:32:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51300K .......... .......... .......... .......... .......... 30% 50.2M 4s 2024-09-17T23:32:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51350K .......... .......... .......... .......... .......... 30% 23.4M 4s 2024-09-17T23:32:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51400K .......... .......... .......... .......... .......... 30% 40.8M 4s 2024-09-17T23:32:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51450K .......... .......... .......... .......... .......... 30% 22.8M 4s 2024-09-17T23:32:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51500K .......... .......... .......... .......... .......... 30% 56.7M 4s 2024-09-17T23:32:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51550K .......... .......... .......... .......... .......... 30% 14.6M 4s 2024-09-17T23:32:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51600K .......... .......... .......... .......... .......... 30% 58.7M 4s 2024-09-17T23:32:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51650K .......... .......... .......... .......... .......... 30% 33.9M 4s 2024-09-17T23:32:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51700K .......... .......... .......... .......... .......... 30% 24.7M 4s 2024-09-17T23:32:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51750K .......... .......... .......... .......... .......... 30% 24.3M 4s 2024-09-17T23:32:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51800K .......... .......... .......... .......... .......... 30% 58.2M 4s 2024-09-17T23:32:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51850K .......... .......... .......... .......... .......... 30% 24.1M 4s 2024-09-17T23:32:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51900K .......... .......... .......... .......... .......... 30% 38.0M 4s 2024-09-17T23:32:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 51950K .......... .......... .......... .......... .......... 30% 23.5M 4s 2024-09-17T23:32:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52000K .......... .......... .......... .......... .......... 30% 23.3M 4s 2024-09-17T23:32:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52050K .......... .......... .......... .......... .......... 30% 61.2M 4s 2024-09-17T23:32:49.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52100K .......... .......... .......... .......... .......... 30% 23.3M 4s 2024-09-17T23:32:49.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52150K .......... .......... .......... .......... .......... 30% 37.5M 4s 2024-09-17T23:32:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52200K .......... .......... .......... .......... .......... 30% 23.9M 4s 2024-09-17T23:32:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52250K .......... .......... .......... .......... .......... 31% 22.0M 4s 2024-09-17T23:32:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52300K .......... .......... .......... .......... .......... 31% 63.5M 4s 2024-09-17T23:32:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52350K .......... .......... .......... .......... .......... 31% 23.2M 4s 2024-09-17T23:32:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52400K .......... .......... .......... .......... .......... 31% 24.9M 4s 2024-09-17T23:32:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52450K .......... .......... .......... .......... .......... 31% 33.1M 4s 2024-09-17T23:32:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52500K .......... .......... .......... .......... .......... 31% 24.9M 4s 2024-09-17T23:32:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52550K .......... .......... .......... .......... .......... 31% 53.8M 4s 2024-09-17T23:32:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52600K .......... .......... .......... .......... .......... 31% 24.5M 4s 2024-09-17T23:32:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52650K .......... .......... .......... .......... .......... 31% 39.0M 4s 2024-09-17T23:32:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52700K .......... .......... .......... .......... .......... 31% 23.1M 4s 2024-09-17T23:32:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52750K .......... .......... .......... .......... .......... 31% 22.5M 4s 2024-09-17T23:32:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52800K .......... .......... .......... .......... .......... 31% 25.6M 4s 2024-09-17T23:32:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52850K .......... .......... .......... .......... .......... 31% 50.6M 4s 2024-09-17T23:32:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52900K .......... .......... .......... .......... .......... 31% 23.4M 4s 2024-09-17T23:32:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 52950K .......... .......... .......... .......... .......... 31% 36.5M 4s 2024-09-17T23:32:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53000K .......... .......... .......... .......... .......... 31% 23.6M 4s 2024-09-17T23:32:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53050K .......... .......... .......... .......... .......... 31% 51.8M 4s 2024-09-17T23:32:49.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53100K .......... .......... .......... .......... .......... 31% 23.5M 4s 2024-09-17T23:32:49.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53150K .......... .......... .......... .......... .......... 31% 23.2M 4s 2024-09-17T23:32:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53200K .......... .......... .......... .......... .......... 31% 42.0M 4s 2024-09-17T23:32:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53250K .......... .......... .......... .......... .......... 31% 22.8M 4s 2024-09-17T23:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53300K .......... .......... .......... .......... .......... 31% 23.1M 4s 2024-09-17T23:32:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53350K .......... .......... .......... .......... .......... 31% 89.8M 4s 2024-09-17T23:32:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53400K .......... .......... .......... .......... .......... 31% 20.6M 4s 2024-09-17T23:32:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53450K .......... .......... .......... .......... .......... 31% 37.6M 4s 2024-09-17T23:32:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53500K .......... .......... .......... .......... .......... 31% 22.8M 4s 2024-09-17T23:32:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53550K .......... .......... .......... .......... .......... 31% 25.2M 4s 2024-09-17T23:32:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53600K .......... .......... .......... .......... .......... 31% 46.5M 4s 2024-09-17T23:32:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53650K .......... .......... .......... .......... .......... 31% 23.4M 4s 2024-09-17T23:32:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53700K .......... .......... .......... .......... .......... 31% 37.4M 4s 2024-09-17T23:32:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53750K .......... .......... .......... .......... .......... 31% 23.7M 4s 2024-09-17T23:32:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53800K .......... .......... .......... .......... .......... 31% 52.9M 4s 2024-09-17T23:32:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53850K .......... .......... .......... .......... .......... 31% 26.3M 4s 2024-09-17T23:32:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53900K .......... .......... .......... .......... .......... 32% 35.5M 4s 2024-09-17T23:32:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 53950K .......... .......... .......... .......... .......... 32% 17.0M 4s 2024-09-17T23:32:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54000K .......... .......... .......... .......... .......... 32% 38.0M 4s 2024-09-17T23:32:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54050K .......... .......... .......... .......... .......... 32% 23.3M 4s 2024-09-17T23:32:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54100K .......... .......... .......... .......... .......... 32% 63.6M 4s 2024-09-17T23:32:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54150K .......... .......... .......... .......... .......... 32% 21.2M 4s 2024-09-17T23:32:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54200K .......... .......... .......... .......... .......... 32% 41.6M 4s 2024-09-17T23:32:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54250K .......... .......... .......... .......... .......... 32% 24.1M 4s 2024-09-17T23:32:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54300K .......... .......... .......... .......... .......... 32% 44.6M 4s 2024-09-17T23:32:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54350K .......... .......... .......... .......... .......... 32% 23.8M 4s 2024-09-17T23:32:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54400K .......... .......... .......... .......... .......... 32% 25.0M 4s 2024-09-17T23:32:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54450K .......... .......... .......... .......... .......... 32% 35.2M 4s 2024-09-17T23:32:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54500K .......... .......... .......... .......... .......... 32% 26.0M 4s 2024-09-17T23:32:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54550K .......... .......... .......... .......... .......... 32% 22.9M 4s 2024-09-17T23:32:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54600K .......... .......... .......... .......... .......... 32% 48.4M 4s 2024-09-17T23:32:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54650K .......... .......... .......... .......... .......... 32% 24.2M 4s 2024-09-17T23:32:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54700K .......... .......... .......... .......... .......... 32% 36.7M 4s 2024-09-17T23:32:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54750K .......... .......... .......... .......... .......... 32% 25.5M 4s 2024-09-17T23:32:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54800K .......... .......... .......... .......... .......... 32% 23.6M 4s 2024-09-17T23:32:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54850K .......... .......... .......... .......... .......... 32% 46.2M 4s 2024-09-17T23:32:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54900K .......... .......... .......... .......... .......... 32% 24.3M 4s 2024-09-17T23:32:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 54950K .......... .......... .......... .......... .......... 32% 35.5M 4s 2024-09-17T23:32:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55000K .......... .......... .......... .......... .......... 32% 22.7M 4s 2024-09-17T23:32:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55050K .......... .......... .......... .......... .......... 32% 25.4M 4s 2024-09-17T23:32:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55100K .......... .......... .......... .......... .......... 32% 57.5M 4s 2024-09-17T23:32:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55150K .......... .......... .......... .......... .......... 32% 23.0M 4s 2024-09-17T23:32:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55200K .......... .......... .......... .......... .......... 32% 23.9M 4s 2024-09-17T23:32:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55250K .......... .......... .......... .......... .......... 32% 34.9M 4s 2024-09-17T23:32:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55300K .......... .......... .......... .......... .......... 32% 25.8M 4s 2024-09-17T23:32:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55350K .......... .......... .......... .......... .......... 32% 61.6M 4s 2024-09-17T23:32:49.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55400K .......... .......... .......... .......... .......... 32% 22.1M 4s 2024-09-17T23:32:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55450K .......... .......... .......... .......... .......... 32% 35.3M 4s 2024-09-17T23:32:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55500K .......... .......... .......... .......... .......... 32% 25.8M 4s 2024-09-17T23:32:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55550K .......... .......... .......... .......... .......... 32% 12.6M 4s 2024-09-17T23:32:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55600K .......... .......... .......... .......... .......... 33% 168M 4s 2024-09-17T23:32:49.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55650K .......... .......... .......... .......... .......... 33% 41.5M 4s 2024-09-17T23:32:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55700K .......... .......... .......... .......... .......... 33% 25.8M 4s 2024-09-17T23:32:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55750K .......... .......... .......... .......... .......... 33% 40.1M 4s 2024-09-17T23:32:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55800K .......... .......... .......... .......... .......... 33% 22.8M 4s 2024-09-17T23:32:49.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55850K .......... .......... .......... .......... .......... 33% 55.7M 4s 2024-09-17T23:32:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55900K .......... .......... .......... .......... .......... 33% 23.0M 4s 2024-09-17T23:32:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 55950K .......... .......... .......... .......... .......... 33% 23.7M 4s 2024-09-17T23:32:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56000K .......... .......... .......... .......... .......... 33% 44.1M 4s 2024-09-17T23:32:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56050K .......... .......... .......... .......... .......... 33% 22.8M 4s 2024-09-17T23:32:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56100K .......... .......... .......... .......... .......... 33% 23.3M 4s 2024-09-17T23:32:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56150K .......... .......... .......... .......... .......... 33% 53.0M 4s 2024-09-17T23:32:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56200K .......... .......... .......... .......... .......... 33% 36.1M 4s 2024-09-17T23:32:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56250K .......... .......... .......... .......... .......... 33% 23.2M 4s 2024-09-17T23:32:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56300K .......... .......... .......... .......... .......... 33% 26.2M 4s 2024-09-17T23:32:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56350K .......... .......... .......... .......... .......... 33% 40.0M 4s 2024-09-17T23:32:49.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56400K .......... .......... .......... .......... .......... 33% 27.4M 4s 2024-09-17T23:32:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56450K .......... .......... .......... .......... .......... 33% 24.2M 4s 2024-09-17T23:32:49.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56500K .......... .......... .......... .......... .......... 33% 38.3M 4s 2024-09-17T23:32:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56550K .......... .......... .......... .......... .......... 33% 24.2M 4s 2024-09-17T23:32:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56600K .......... .......... .......... .......... .......... 33% 42.1M 4s 2024-09-17T23:32:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56650K .......... .......... .......... .......... .......... 33% 24.3M 4s 2024-09-17T23:32:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56700K .......... .......... .......... .......... .......... 33% 24.8M 4s 2024-09-17T23:32:49.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56750K .......... .......... .......... .......... .......... 33% 23.0M 4s 2024-09-17T23:32:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56800K .......... .......... .......... .......... .......... 33% 31.2M 4s 2024-09-17T23:32:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56850K .......... .......... .......... .......... .......... 33% 56.8M 4s 2024-09-17T23:32:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56900K .......... .......... .......... .......... .......... 33% 29.2M 4s 2024-09-17T23:32:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 56950K .......... .......... .......... .......... .......... 33% 21.3M 4s 2024-09-17T23:32:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57000K .......... .......... .......... .......... .......... 33% 46.2M 4s 2024-09-17T23:32:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57050K .......... .......... .......... .......... .......... 33% 22.9M 4s 2024-09-17T23:32:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57100K .......... .......... .......... .......... .......... 33% 49.8M 4s 2024-09-17T23:32:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57150K .......... .......... .......... .......... .......... 33% 24.7M 4s 2024-09-17T23:32:49.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57200K .......... .......... .......... .......... .......... 33% 23.3M 4s 2024-09-17T23:32:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57250K .......... .......... .......... .......... .......... 33% 32.7M 4s 2024-09-17T23:32:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57300K .......... .......... .......... .......... .......... 34% 24.6M 4s 2024-09-17T23:32:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57350K .......... .......... .......... .......... .......... 34% 25.1M 4s 2024-09-17T23:32:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57400K .......... .......... .......... .......... .......... 34% 46.7M 4s 2024-09-17T23:32:49.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57450K .......... .......... .......... .......... .......... 34% 25.7M 4s 2024-09-17T23:32:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57500K .......... .......... .......... .......... .......... 34% 41.7M 4s 2024-09-17T23:32:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57550K .......... .......... .......... .......... .......... 34% 22.6M 4s 2024-09-17T23:32:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57600K .......... .......... .......... .......... .......... 34% 24.3M 4s 2024-09-17T23:32:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57650K .......... .......... .......... .......... .......... 34% 45.2M 4s 2024-09-17T23:32:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57700K .......... .......... .......... .......... .......... 34% 25.0M 4s 2024-09-17T23:32:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57750K .......... .......... .......... .......... .......... 34% 40.0M 4s 2024-09-17T23:32:49.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57800K .......... .......... .......... .......... .......... 34% 23.5M 4s 2024-09-17T23:32:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57850K .......... .......... .......... .......... .......... 34% 22.9M 4s 2024-09-17T23:32:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57900K .......... .......... .......... .......... .......... 34% 50.9M 4s 2024-09-17T23:32:49.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 57950K .......... .......... .......... .......... .......... 34% 21.7M 4s 2024-09-17T23:32:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58000K .......... .......... .......... .......... .......... 34% 25.6M 4s 2024-09-17T23:32:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58050K .......... .......... .......... .......... .......... 34% 38.1M 4s 2024-09-17T23:32:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58100K .......... .......... .......... .......... .......... 34% 23.7M 4s 2024-09-17T23:32:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58150K .......... .......... .......... .......... .......... 34% 52.7M 4s 2024-09-17T23:32:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58200K .......... .......... .......... .......... .......... 34% 24.0M 4s 2024-09-17T23:32:49.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58250K .......... .......... .......... .......... .......... 34% 28.0M 4s 2024-09-17T23:32:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58300K .......... .......... .......... .......... .......... 34% 30.4M 4s 2024-09-17T23:32:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58350K .......... .......... .......... .......... .......... 34% 23.2M 4s 2024-09-17T23:32:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58400K .......... .......... .......... .......... .......... 34% 24.5M 4s 2024-09-17T23:32:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58450K .......... .......... .......... .......... .......... 34% 51.8M 4s 2024-09-17T23:32:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58500K .......... .......... .......... .......... .......... 34% 23.4M 4s 2024-09-17T23:32:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58550K .......... .......... .......... .......... .......... 34% 31.0M 4s 2024-09-17T23:32:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58600K .......... .......... .......... .......... .......... 34% 29.8M 4s 2024-09-17T23:32:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58650K .......... .......... .......... .......... .......... 34% 47.7M 4s 2024-09-17T23:32:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58700K .......... .......... .......... .......... .......... 34% 24.0M 4s 2024-09-17T23:32:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58750K .......... .......... .......... .......... .......... 34% 23.8M 4s 2024-09-17T23:32:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58800K .......... .......... .......... .......... .......... 34% 41.9M 4s 2024-09-17T23:32:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58850K .......... .......... .......... .......... .......... 34% 23.8M 4s 2024-09-17T23:32:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58900K .......... .......... .......... .......... .......... 34% 28.3M 4s 2024-09-17T23:32:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 58950K .......... .......... .......... .......... .......... 35% 41.4M 4s 2024-09-17T23:32:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59000K .......... .......... .......... .......... .......... 35% 30.1M 4s 2024-09-17T23:32:49.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59050K .......... .......... .......... .......... .......... 35% 26.8M 4s 2024-09-17T23:32:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59100K .......... .......... .......... .......... .......... 35% 26.2M 4s 2024-09-17T23:32:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59150K .......... .......... .......... .......... .......... 35% 21.5M 4s 2024-09-17T23:32:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59200K .......... .......... .......... .......... .......... 35% 49.8M 4s 2024-09-17T23:32:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59250K .......... .......... .......... .......... .......... 35% 23.1M 4s 2024-09-17T23:32:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59300K .......... .......... .......... .......... .......... 35% 41.2M 4s 2024-09-17T23:32:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59350K .......... .......... .......... .......... .......... 35% 24.4M 4s 2024-09-17T23:32:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59400K .......... .......... .......... .......... .......... 35% 48.5M 4s 2024-09-17T23:32:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59450K .......... .......... .......... .......... .......... 35% 24.5M 4s 2024-09-17T23:32:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59500K .......... .......... .......... .......... .......... 35% 35.0M 4s 2024-09-17T23:32:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59550K .......... .......... .......... .......... .......... 35% 21.4M 4s 2024-09-17T23:32:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59600K .......... .......... .......... .......... .......... 35% 23.5M 4s 2024-09-17T23:32:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59650K .......... .......... .......... .......... .......... 35% 38.2M 4s 2024-09-17T23:32:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59700K .......... .......... .......... .......... .......... 35% 37.1M 4s 2024-09-17T23:32:49.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59750K .......... .......... .......... .......... .......... 35% 21.2M 4s 2024-09-17T23:32:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59800K .......... .......... .......... .......... .......... 35% 42.0M 4s 2024-09-17T23:32:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59850K .......... .......... .......... .......... .......... 35% 23.6M 4s 2024-09-17T23:32:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59900K .......... .......... .......... .......... .......... 35% 50.5M 4s 2024-09-17T23:32:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 59950K .......... .......... .......... .......... .......... 35% 22.7M 4s 2024-09-17T23:32:49.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60000K .......... .......... .......... .......... .......... 35% 25.1M 4s 2024-09-17T23:32:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60050K .......... .......... .......... .......... .......... 35% 38.7M 4s 2024-09-17T23:32:49.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60100K .......... .......... .......... .......... .......... 35% 24.0M 4s 2024-09-17T23:32:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60150K .......... .......... .......... .......... .......... 35% 22.7M 4s 2024-09-17T23:32:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60200K .......... .......... .......... .......... .......... 35% 53.6M 4s 2024-09-17T23:32:49.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60250K .......... .......... .......... .......... .......... 35% 24.5M 4s 2024-09-17T23:32:49.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60300K .......... .......... .......... .......... .......... 35% 38.6M 4s 2024-09-17T23:32:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60350K .......... .......... .......... .......... .......... 35% 24.3M 4s 2024-09-17T23:32:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60400K .......... .......... .......... .......... .......... 35% 23.8M 4s 2024-09-17T23:32:49.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60450K .......... .......... .......... .......... .......... 35% 50.5M 4s 2024-09-17T23:32:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60500K .......... .......... .......... .......... .......... 35% 24.0M 4s 2024-09-17T23:32:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60550K .......... .......... .......... .......... .......... 35% 39.9M 4s 2024-09-17T23:32:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60600K .......... .......... .......... .......... .......... 35% 22.6M 4s 2024-09-17T23:32:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60650K .......... .......... .......... .......... .......... 36% 23.0M 4s 2024-09-17T23:32:49.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60700K .......... .......... .......... .......... .......... 36% 57.7M 4s 2024-09-17T23:32:49.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60750K .......... .......... .......... .......... .......... 36% 24.0M 4s 2024-09-17T23:32:49.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60800K .......... .......... .......... .......... .......... 36% 23.1M 4s 2024-09-17T23:32:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60850K .......... .......... .......... .......... .......... 36% 41.1M 4s 2024-09-17T23:32:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60900K .......... .......... .......... .......... .......... 36% 23.5M 4s 2024-09-17T23:32:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 60950K .......... .......... .......... .......... .......... 36% 53.9M 4s 2024-09-17T23:32:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61000K .......... .......... .......... .......... .......... 36% 22.6M 4s 2024-09-17T23:32:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61050K .......... .......... .......... .......... .......... 36% 39.6M 4s 2024-09-17T23:32:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61100K .......... .......... .......... .......... .......... 36% 29.1M 4s 2024-09-17T23:32:49.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61150K .......... .......... .......... .......... .......... 36% 20.1M 4s 2024-09-17T23:32:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61200K .......... .......... .......... .......... .......... 36% 24.0M 4s 2024-09-17T23:32:49.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61250K .......... .......... .......... .......... .......... 36% 53.1M 4s 2024-09-17T23:32:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61300K .......... .......... .......... .......... .......... 36% 35.6M 4s 2024-09-17T23:32:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61350K .......... .......... .......... .......... .......... 36% 23.1M 4s 2024-09-17T23:32:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61400K .......... .......... .......... .......... .......... 36% 24.1M 4s 2024-09-17T23:32:49.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61450K .......... .......... .......... .......... .......... 36% 56.6M 4s 2024-09-17T23:32:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61500K .......... .......... .......... .......... .......... 36% 23.4M 4s 2024-09-17T23:32:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61550K .......... .......... .......... .......... .......... 36% 25.0M 4s 2024-09-17T23:32:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61600K .......... .......... .......... .......... .......... 36% 34.9M 4s 2024-09-17T23:32:49.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61650K .......... .......... .......... .......... .......... 36% 26.3M 4s 2024-09-17T23:32:49.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61700K .......... .......... .......... .......... .......... 36% 50.0M 4s 2024-09-17T23:32:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61750K .......... .......... .......... .......... .......... 36% 23.6M 4s 2024-09-17T23:32:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61800K .......... .......... .......... .......... .......... 36% 39.1M 4s 2024-09-17T23:32:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61850K .......... .......... .......... .......... .......... 36% 23.7M 4s 2024-09-17T23:32:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61900K .......... .......... .......... .......... .......... 36% 23.9M 4s 2024-09-17T23:32:49.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 61950K .......... .......... .......... .......... .......... 36% 22.5M 4s 2024-09-17T23:32:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62000K .......... .......... .......... .......... .......... 36% 52.3M 4s 2024-09-17T23:32:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62050K .......... .......... .......... .......... .......... 36% 24.2M 4s 2024-09-17T23:32:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62100K .......... .......... .......... .......... .......... 36% 34.8M 4s 2024-09-17T23:32:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62150K .......... .......... .......... .......... .......... 36% 25.1M 4s 2024-09-17T23:32:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62200K .......... .......... .......... .......... .......... 36% 40.8M 4s 2024-09-17T23:32:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62250K .......... .......... .......... .......... .......... 36% 26.3M 4s 2024-09-17T23:32:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62300K .......... .......... .......... .......... .......... 36% 41.7M 4s 2024-09-17T23:32:49.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62350K .......... .......... .......... .......... .......... 37% 21.2M 4s 2024-09-17T23:32:49.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62400K .......... .......... .......... .......... .......... 37% 25.3M 4s 2024-09-17T23:32:49.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62450K .......... .......... .......... .......... .......... 37% 25.1M 4s 2024-09-17T23:32:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62500K .......... .......... .......... .......... .......... 37% 51.6M 4s 2024-09-17T23:32:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62550K .......... .......... .......... .......... .......... 37% 22.8M 4s 2024-09-17T23:32:49.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62600K .......... .......... .......... .......... .......... 37% 37.1M 4s 2024-09-17T23:32:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62650K .......... .......... .......... .......... .......... 37% 22.9M 4s 2024-09-17T23:32:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62700K .......... .......... .......... .......... .......... 37% 58.2M 4s 2024-09-17T23:32:49.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62750K .......... .......... .......... .......... .......... 37% 24.0M 4s 2024-09-17T23:32:49.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62800K .......... .......... .......... .......... .......... 37% 23.7M 4s 2024-09-17T23:32:49.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62850K .......... .......... .......... .......... .......... 37% 39.1M 4s 2024-09-17T23:32:49.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62900K .......... .......... .......... .......... .......... 37% 23.5M 4s 2024-09-17T23:32:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 62950K .......... .......... .......... .......... .......... 37% 22.4M 4s 2024-09-17T23:32:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63000K .......... .......... .......... .......... .......... 37% 56.1M 4s 2024-09-17T23:32:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63050K .......... .......... .......... .......... .......... 37% 23.9M 4s 2024-09-17T23:32:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63100K .......... .......... .......... .......... .......... 37% 45.5M 4s 2024-09-17T23:32:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63150K .......... .......... .......... .......... .......... 37% 21.8M 4s 2024-09-17T23:32:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63200K .......... .......... .......... .......... .......... 37% 23.7M 4s 2024-09-17T23:32:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63250K .......... .......... .......... .......... .......... 37% 52.7M 4s 2024-09-17T23:32:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63300K .......... .......... .......... .......... .......... 37% 24.2M 4s 2024-09-17T23:32:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63350K .......... .......... .......... .......... .......... 37% 34.4M 4s 2024-09-17T23:32:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63400K .......... .......... .......... .......... .......... 37% 24.5M 4s 2024-09-17T23:32:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63450K .......... .......... .......... .......... .......... 37% 23.2M 4s 2024-09-17T23:32:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63500K .......... .......... .......... .......... .......... 37% 58.5M 4s 2024-09-17T23:32:49.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63550K .......... .......... .......... .......... .......... 37% 30.0M 4s 2024-09-17T23:32:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63600K .......... .......... .......... .......... .......... 37% 25.2M 4s 2024-09-17T23:32:49.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63650K .......... .......... .......... .......... .......... 37% 24.1M 4s 2024-09-17T23:32:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63700K .......... .......... .......... .......... .......... 37% 26.1M 4s 2024-09-17T23:32:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63750K .......... .......... .......... .......... .......... 37% 63.9M 4s 2024-09-17T23:32:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63800K .......... .......... .......... .......... .......... 37% 23.4M 4s 2024-09-17T23:32:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63850K .......... .......... .......... .......... .......... 37% 35.5M 4s 2024-09-17T23:32:49.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63900K .......... .......... .......... .......... .......... 37% 23.4M 4s 2024-09-17T23:32:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 63950K .......... .......... .......... .......... .......... 37% 22.8M 4s 2024-09-17T23:32:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64000K .......... .......... .......... .......... .......... 37% 23.6M 4s 2024-09-17T23:32:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64050K .......... .......... .......... .......... .......... 38% 155M 4s 2024-09-17T23:32:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64100K .......... .......... .......... .......... .......... 38% 27.1M 4s 2024-09-17T23:32:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64150K .......... .......... .......... .......... .......... 38% 23.4M 4s 2024-09-17T23:32:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64200K .......... .......... .......... .......... .......... 38% 22.9M 4s 2024-09-17T23:32:49.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64250K .......... .......... .......... .......... .......... 38% 56.8M 4s 2024-09-17T23:32:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64300K .......... .......... .......... .......... .......... 38% 23.5M 4s 2024-09-17T23:32:49.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64350K .......... .......... .......... .......... .......... 38% 24.1M 4s 2024-09-17T23:32:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64400K .......... .......... .......... .......... .......... 38% 37.5M 4s 2024-09-17T23:32:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64450K .......... .......... .......... .......... .......... 38% 24.3M 4s 2024-09-17T23:32:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64500K .......... .......... .......... .......... .......... 38% 49.6M 4s 2024-09-17T23:32:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64550K .......... .......... .......... .......... .......... 38% 23.2M 4s 2024-09-17T23:32:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64600K .......... .......... .......... .......... .......... 38% 40.7M 4s 2024-09-17T23:32:49.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64650K .......... .......... .......... .......... .......... 38% 24.5M 4s 2024-09-17T23:32:49.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64700K .......... .......... .......... .......... .......... 38% 23.4M 4s 2024-09-17T23:32:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64750K .......... .......... .......... .......... .......... 38% 23.6M 4s 2024-09-17T23:32:49.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64800K .......... .......... .......... .......... .......... 38% 54.9M 4s 2024-09-17T23:32:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64850K .......... .......... .......... .......... .......... 38% 23.9M 4s 2024-09-17T23:32:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64900K .......... .......... .......... .......... .......... 38% 39.5M 4s 2024-09-17T23:32:49.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 64950K .......... .......... .......... .......... .......... 38% 23.7M 4s 2024-09-17T23:32:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65000K .......... .......... .......... .......... .......... 38% 49.9M 4s 2024-09-17T23:32:49.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65050K .......... .......... .......... .......... .......... 38% 26.6M 4s 2024-09-17T23:32:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65100K .......... .......... .......... .......... .......... 38% 35.2M 4s 2024-09-17T23:32:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65150K .......... .......... .......... .......... .......... 38% 23.9M 4s 2024-09-17T23:32:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65200K .......... .......... .......... .......... .......... 38% 23.2M 4s 2024-09-17T23:32:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65250K .......... .......... .......... .......... .......... 38% 24.3M 4s 2024-09-17T23:32:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65300K .......... .......... .......... .......... .......... 38% 47.5M 4s 2024-09-17T23:32:49.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65350K .......... .......... .......... .......... .......... 38% 23.8M 4s 2024-09-17T23:32:49.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65400K .......... .......... .......... .......... .......... 38% 41.4M 4s 2024-09-17T23:32:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65450K .......... .......... .......... .......... .......... 38% 22.1M 4s 2024-09-17T23:32:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65500K .......... .......... .......... .......... .......... 38% 59.7M 4s 2024-09-17T23:32:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65550K .......... .......... .......... .......... .......... 38% 23.1M 4s 2024-09-17T23:32:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65600K .......... .......... .......... .......... .......... 38% 23.7M 4s 2024-09-17T23:32:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65650K .......... .......... .......... .......... .......... 38% 38.2M 4s 2024-09-17T23:32:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65700K .......... .......... .......... .......... .......... 39% 24.5M 4s 2024-09-17T23:32:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65750K .......... .......... .......... .......... .......... 39% 23.6M 4s 2024-09-17T23:32:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65800K .......... .......... .......... .......... .......... 39% 54.7M 4s 2024-09-17T23:32:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65850K .......... .......... .......... .......... .......... 39% 22.5M 4s 2024-09-17T23:32:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65900K .......... .......... .......... .......... .......... 39% 37.9M 4s 2024-09-17T23:32:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 65950K .......... .......... .......... .......... .......... 39% 24.7M 4s 2024-09-17T23:32:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66000K .......... .......... .......... .......... .......... 39% 21.8M 4s 2024-09-17T23:32:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66050K .......... .......... .......... .......... .......... 39% 52.7M 4s 2024-09-17T23:32:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66100K .......... .......... .......... .......... .......... 39% 24.5M 4s 2024-09-17T23:32:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66150K .......... .......... .......... .......... .......... 39% 38.6M 4s 2024-09-17T23:32:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66200K .......... .......... .......... .......... .......... 39% 22.9M 4s 2024-09-17T23:32:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66250K .......... .......... .......... .......... .......... 39% 54.3M 4s 2024-09-17T23:32:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66300K .......... .......... .......... .......... .......... 39% 24.0M 4s 2024-09-17T23:32:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66350K .......... .......... .......... .......... .......... 39% 25.2M 4s 2024-09-17T23:32:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66400K .......... .......... .......... .......... .......... 39% 30.3M 4s 2024-09-17T23:32:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66450K .......... .......... .......... .......... .......... 39% 25.6M 4s 2024-09-17T23:32:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66500K .......... .......... .......... .......... .......... 39% 23.1M 4s 2024-09-17T23:32:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66550K .......... .......... .......... .......... .......... 39% 55.6M 4s 2024-09-17T23:32:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66600K .......... .......... .......... .......... .......... 39% 23.8M 4s 2024-09-17T23:32:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66650K .......... .......... .......... .......... .......... 39% 38.6M 4s 2024-09-17T23:32:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66700K .......... .......... .......... .......... .......... 39% 23.0M 4s 2024-09-17T23:32:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66750K .......... .......... .......... .......... .......... 39% 23.5M 4s 2024-09-17T23:32:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66800K .......... .......... .......... .......... .......... 39% 59.1M 4s 2024-09-17T23:32:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66850K .......... .......... .......... .......... .......... 39% 24.5M 4s 2024-09-17T23:32:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66900K .......... .......... .......... .......... .......... 39% 37.3M 4s 2024-09-17T23:32:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 66950K .......... .......... .......... .......... .......... 39% 23.9M 4s 2024-09-17T23:32:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67000K .......... .......... .......... .......... .......... 39% 24.3M 4s 2024-09-17T23:32:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67050K .......... .......... .......... .......... .......... 39% 47.4M 4s 2024-09-17T23:32:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67100K .......... .......... .......... .......... .......... 39% 23.0M 4s 2024-09-17T23:32:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67150K .......... .......... .......... .......... .......... 39% 23.7M 4s 2024-09-17T23:32:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67200K .......... .......... .......... .......... .......... 39% 40.7M 4s 2024-09-17T23:32:49.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67250K .......... .......... .......... .......... .......... 39% 23.3M 4s 2024-09-17T23:32:49.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67300K .......... .......... .......... .......... .......... 39% 57.1M 4s 2024-09-17T23:32:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67350K .......... .......... .......... .......... .......... 39% 23.7M 4s 2024-09-17T23:32:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67400K .......... .......... .......... .......... .......... 40% 38.4M 3s 2024-09-17T23:32:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67450K .......... .......... .......... .......... .......... 40% 22.8M 3s 2024-09-17T23:32:49.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67500K .......... .......... .......... .......... .......... 40% 22.3M 3s 2024-09-17T23:32:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67550K .......... .......... .......... .......... .......... 40% 23.1M 3s 2024-09-17T23:32:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67600K .......... .......... .......... .......... .......... 40% 76.5M 3s 2024-09-17T23:32:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67650K .......... .......... .......... .......... .......... 40% 33.6M 3s 2024-09-17T23:32:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67700K .......... .......... .......... .......... .......... 40% 23.7M 3s 2024-09-17T23:32:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67750K .......... .......... .......... .......... .......... 40% 23.1M 3s 2024-09-17T23:32:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67800K .......... .......... .......... .......... .......... 40% 57.6M 3s 2024-09-17T23:32:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67850K .......... .......... .......... .......... .......... 40% 23.9M 3s 2024-09-17T23:32:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67900K .......... .......... .......... .......... .......... 40% 45.0M 3s 2024-09-17T23:32:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 67950K .......... .......... .......... .......... .......... 40% 17.2M 3s 2024-09-17T23:32:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68000K .......... .......... .......... .......... .......... 40% 31.9M 3s 2024-09-17T23:32:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68050K .......... .......... .......... .......... .......... 40% 25.8M 3s 2024-09-17T23:32:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68100K .......... .......... .......... .......... .......... 40% 55.9M 3s 2024-09-17T23:32:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68150K .......... .......... .......... .......... .......... 40% 22.8M 3s 2024-09-17T23:32:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68200K .......... .......... .......... .......... .......... 40% 33.0M 3s 2024-09-17T23:32:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68250K .......... .......... .......... .......... .......... 40% 38.3M 3s 2024-09-17T23:32:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68300K .......... .......... .......... .......... .......... 40% 35.7M 3s 2024-09-17T23:32:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68350K .......... .......... .......... .......... .......... 40% 23.1M 3s 2024-09-17T23:32:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68400K .......... .......... .......... .......... .......... 40% 23.6M 3s 2024-09-17T23:32:49.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68450K .......... .......... .......... .......... .......... 40% 38.8M 3s 2024-09-17T23:32:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68500K .......... .......... .......... .......... .......... 40% 24.9M 3s 2024-09-17T23:32:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68550K .......... .......... .......... .......... .......... 40% 26.0M 3s 2024-09-17T23:32:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68600K .......... .......... .......... .......... .......... 40% 38.3M 3s 2024-09-17T23:32:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68650K .......... .......... .......... .......... .......... 40% 36.9M 3s 2024-09-17T23:32:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68700K .......... .......... .......... .......... .......... 40% 22.6M 3s 2024-09-17T23:32:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68750K .......... .......... .......... .......... .......... 40% 26.5M 3s 2024-09-17T23:32:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68800K .......... .......... .......... .......... .......... 40% 24.0M 3s 2024-09-17T23:32:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68850K .......... .......... .......... .......... .......... 40% 50.2M 3s 2024-09-17T23:32:50.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68900K .......... .......... .......... .......... .......... 40% 26.7M 3s 2024-09-17T23:32:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 68950K .......... .......... .......... .......... .......... 40% 25.7M 3s 2024-09-17T23:32:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69000K .......... .......... .......... .......... .......... 40% 23.2M 3s 2024-09-17T23:32:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69050K .......... .......... .......... .......... .......... 40% 122M 3s 2024-09-17T23:32:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69100K .......... .......... .......... .......... .......... 41% 22.3M 3s 2024-09-17T23:32:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69150K .......... .......... .......... .......... .......... 41% 24.4M 3s 2024-09-17T23:32:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69200K .......... .......... .......... .......... .......... 41% 23.2M 3s 2024-09-17T23:32:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69250K .......... .......... .......... .......... .......... 41% 37.1M 3s 2024-09-17T23:32:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69300K .......... .......... .......... .......... .......... 41% 24.7M 3s 2024-09-17T23:32:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69350K .......... .......... .......... .......... .......... 41% 47.0M 3s 2024-09-17T23:32:50.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69400K .......... .......... .......... .......... .......... 41% 24.2M 3s 2024-09-17T23:32:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69450K .......... .......... .......... .......... .......... 41% 34.2M 3s 2024-09-17T23:32:50.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69500K .......... .......... .......... .......... .......... 41% 26.4M 3s 2024-09-17T23:32:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69550K .......... .......... .......... .......... .......... 41% 23.2M 3s 2024-09-17T23:32:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69600K .......... .......... .......... .......... .......... 41% 55.8M 3s 2024-09-17T23:32:50.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69650K .......... .......... .......... .......... .......... 41% 25.4M 3s 2024-09-17T23:32:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69700K .......... .......... .......... .......... .......... 41% 23.5M 3s 2024-09-17T23:32:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69750K .......... .......... .......... .......... .......... 41% 31.4M 3s 2024-09-17T23:32:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69800K .......... .......... .......... .......... .......... 41% 42.0M 3s 2024-09-17T23:32:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69850K .......... .......... .......... .......... .......... 41% 26.4M 3s 2024-09-17T23:32:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69900K .......... .......... .......... .......... .......... 41% 20.2M 3s 2024-09-17T23:32:50.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 69950K .......... .......... .......... .......... .......... 41% 29.5M 3s 2024-09-17T23:32:50.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70000K .......... .......... .......... .......... .......... 41% 38.6M 3s 2024-09-17T23:32:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70050K .......... .......... .......... .......... .......... 41% 25.1M 3s 2024-09-17T23:32:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70100K .......... .......... .......... .......... .......... 41% 48.4M 3s 2024-09-17T23:32:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70150K .......... .......... .......... .......... .......... 41% 23.8M 3s 2024-09-17T23:32:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70200K .......... .......... .......... .......... .......... 41% 24.4M 3s 2024-09-17T23:32:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70250K .......... .......... .......... .......... .......... 41% 37.9M 3s 2024-09-17T23:32:50.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70300K .......... .......... .......... .......... .......... 41% 22.8M 3s 2024-09-17T23:32:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70350K .......... .......... .......... .......... .......... 41% 20.7M 3s 2024-09-17T23:32:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70400K .......... .......... .......... .......... .......... 41% 78.0M 3s 2024-09-17T23:32:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70450K .......... .......... .......... .......... .......... 41% 24.6M 3s 2024-09-17T23:32:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70500K .......... .......... .......... .......... .......... 41% 36.2M 3s 2024-09-17T23:32:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70550K .......... .......... .......... .......... .......... 41% 23.8M 3s 2024-09-17T23:32:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70600K .......... .......... .......... .......... .......... 41% 51.1M 3s 2024-09-17T23:32:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70650K .......... .......... .......... .......... .......... 41% 23.9M 3s 2024-09-17T23:32:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70700K .......... .......... .......... .......... .......... 41% 37.4M 3s 2024-09-17T23:32:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70750K .......... .......... .......... .......... .......... 42% 25.0M 3s 2024-09-17T23:32:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70800K .......... .......... .......... .......... .......... 42% 27.2M 3s 2024-09-17T23:32:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70850K .......... .......... .......... .......... .......... 42% 20.0M 3s 2024-09-17T23:32:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70900K .......... .......... .......... .......... .......... 42% 58.7M 3s 2024-09-17T23:32:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 70950K .......... .......... .......... .......... .......... 42% 23.2M 3s 2024-09-17T23:32:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71000K .......... .......... .......... .......... .......... 42% 38.7M 3s 2024-09-17T23:32:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71050K .......... .......... .......... .......... .......... 42% 22.5M 3s 2024-09-17T23:32:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71100K .......... .......... .......... .......... .......... 42% 57.5M 3s 2024-09-17T23:32:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71150K .......... .......... .......... .......... .......... 42% 23.8M 3s 2024-09-17T23:32:50.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71200K .......... .......... .......... .......... .......... 42% 24.3M 3s 2024-09-17T23:32:50.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71250K .......... .......... .......... .......... .......... 42% 25.5M 3s 2024-09-17T23:32:50.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71300K .......... .......... .......... .......... .......... 42% 34.6M 3s 2024-09-17T23:32:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71350K .......... .......... .......... .......... .......... 42% 45.7M 3s 2024-09-17T23:32:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71400K .......... .......... .......... .......... .......... 42% 24.2M 3s 2024-09-17T23:32:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71450K .......... .......... .......... .......... .......... 42% 31.9M 3s 2024-09-17T23:32:50.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71500K .......... .......... .......... .......... .......... 42% 27.4M 3s 2024-09-17T23:32:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71550K .......... .......... .......... .......... .......... 42% 23.8M 3s 2024-09-17T23:32:50.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71600K .......... .......... .......... .......... .......... 42% 23.2M 3s 2024-09-17T23:32:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71650K .......... .......... .......... .......... .......... 42% 54.7M 3s 2024-09-17T23:32:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71700K .......... .......... .......... .......... .......... 42% 23.4M 3s 2024-09-17T23:32:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71750K .......... .......... .......... .......... .......... 42% 39.0M 3s 2024-09-17T23:32:50.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71800K .......... .......... .......... .......... .......... 42% 23.1M 3s 2024-09-17T23:32:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71850K .......... .......... .......... .......... .......... 42% 55.7M 3s 2024-09-17T23:32:50.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71900K .......... .......... .......... .......... .......... 42% 23.5M 3s 2024-09-17T23:32:50.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 71950K .......... .......... .......... .......... .......... 42% 23.8M 3s 2024-09-17T23:32:50.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72000K .......... .......... .......... .......... .......... 42% 37.2M 3s 2024-09-17T23:32:50.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72050K .......... .......... .......... .......... .......... 42% 23.9M 3s 2024-09-17T23:32:50.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72100K .......... .......... .......... .......... .......... 42% 24.6M 3s 2024-09-17T23:32:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72150K .......... .......... .......... .......... .......... 42% 47.7M 3s 2024-09-17T23:32:50.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72200K .......... .......... .......... .......... .......... 42% 24.0M 3s 2024-09-17T23:32:50.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72250K .......... .......... .......... .......... .......... 42% 38.6M 3s 2024-09-17T23:32:50.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72300K .......... .......... .......... .......... .......... 42% 23.3M 3s 2024-09-17T23:32:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72350K .......... .......... .......... .......... .......... 42% 24.8M 3s 2024-09-17T23:32:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72400K .......... .......... .......... .......... .......... 42% 43.9M 3s 2024-09-17T23:32:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72450K .......... .......... .......... .......... .......... 43% 24.8M 3s 2024-09-17T23:32:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72500K .......... .......... .......... .......... .......... 43% 41.6M 3s 2024-09-17T23:32:50.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72550K .......... .......... .......... .......... .......... 43% 21.5M 3s 2024-09-17T23:32:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72600K .......... .......... .......... .......... .......... 43% 24.3M 3s 2024-09-17T23:32:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72650K .......... .......... .......... .......... .......... 43% 59.8M 3s 2024-09-17T23:32:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72700K .......... .......... .......... .......... .......... 43% 25.0M 3s 2024-09-17T23:32:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72750K .......... .......... .......... .......... .......... 43% 22.0M 3s 2024-09-17T23:32:50.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72800K .......... .......... .......... .......... .......... 43% 40.9M 3s 2024-09-17T23:32:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72850K .......... .......... .......... .......... .......... 43% 23.1M 3s 2024-09-17T23:32:50.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72900K .......... .......... .......... .......... .......... 43% 54.9M 3s 2024-09-17T23:32:50.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 72950K .......... .......... .......... .......... .......... 43% 22.6M 3s 2024-09-17T23:32:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73000K .......... .......... .......... .......... .......... 43% 39.2M 3s 2024-09-17T23:32:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73050K .......... .......... .......... .......... .......... 43% 24.8M 3s 2024-09-17T23:32:50.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73100K .......... .......... .......... .......... .......... 43% 23.4M 3s 2024-09-17T23:32:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73150K .......... .......... .......... .......... .......... 43% 23.2M 3s 2024-09-17T23:32:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73200K .......... .......... .......... .......... .......... 43% 53.5M 3s 2024-09-17T23:32:50.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73250K .......... .......... .......... .......... .......... 43% 24.1M 3s 2024-09-17T23:32:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73300K .......... .......... .......... .......... .......... 43% 34.4M 3s 2024-09-17T23:32:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73350K .......... .......... .......... .......... .......... 43% 29.3M 3s 2024-09-17T23:32:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73400K .......... .......... .......... .......... .......... 43% 37.2M 3s 2024-09-17T23:32:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73450K .......... .......... .......... .......... .......... 43% 25.7M 3s 2024-09-17T23:32:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73500K .......... .......... .......... .......... .......... 43% 43.0M 3s 2024-09-17T23:32:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73550K .......... .......... .......... .......... .......... 43% 22.8M 3s 2024-09-17T23:32:50.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73600K .......... .......... .......... .......... .......... 43% 23.0M 3s 2024-09-17T23:32:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73650K .......... .......... .......... .......... .......... 43% 24.5M 3s 2024-09-17T23:32:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73700K .......... .......... .......... .......... .......... 43% 54.4M 3s 2024-09-17T23:32:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73750K .......... .......... .......... .......... .......... 43% 30.5M 3s 2024-09-17T23:32:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73800K .......... .......... .......... .......... .......... 43% 24.8M 3s 2024-09-17T23:32:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73850K .......... .......... .......... .......... .......... 43% 24.5M 3s 2024-09-17T23:32:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73900K .......... .......... .......... .......... .......... 43% 47.4M 3s 2024-09-17T23:32:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 73950K .......... .......... .......... .......... .......... 43% 23.3M 3s 2024-09-17T23:32:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74000K .......... .......... .......... .......... .......... 43% 24.9M 3s 2024-09-17T23:32:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74050K .......... .......... .......... .......... .......... 43% 41.7M 3s 2024-09-17T23:32:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74100K .......... .......... .......... .......... .......... 43% 27.1M 3s 2024-09-17T23:32:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74150K .......... .......... .......... .......... .......... 44% 39.5M 3s 2024-09-17T23:32:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74200K .......... .......... .......... .......... .......... 44% 27.8M 3s 2024-09-17T23:32:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74250K .......... .......... .......... .......... .......... 44% 30.8M 3s 2024-09-17T23:32:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74300K .......... .......... .......... .......... .......... 44% 24.2M 3s 2024-09-17T23:32:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74350K .......... .......... .......... .......... .......... 44% 23.5M 3s 2024-09-17T23:32:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74400K .......... .......... .......... .......... .......... 44% 23.3M 3s 2024-09-17T23:32:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74450K .......... .......... .......... .......... .......... 44% 54.3M 3s 2024-09-17T23:32:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74500K .......... .......... .......... .......... .......... 44% 26.5M 3s 2024-09-17T23:32:50.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74550K .......... .......... .......... .......... .......... 44% 33.9M 3s 2024-09-17T23:32:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74600K .......... .......... .......... .......... .......... 44% 23.7M 3s 2024-09-17T23:32:50.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74650K .......... .......... .......... .......... .......... 44% 54.1M 3s 2024-09-17T23:32:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74700K .......... .......... .......... .......... .......... 44% 23.7M 3s 2024-09-17T23:32:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74750K .......... .......... .......... .......... .......... 44% 23.9M 3s 2024-09-17T23:32:50.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74800K .......... .......... .......... .......... .......... 44% 36.9M 3s 2024-09-17T23:32:50.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74850K .......... .......... .......... .......... .......... 44% 22.6M 3s 2024-09-17T23:32:50.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74900K .......... .......... .......... .......... .......... 44% 26.4M 3s 2024-09-17T23:32:50.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 74950K .......... .......... .......... .......... .......... 44% 67.7M 3s 2024-09-17T23:32:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75000K .......... .......... .......... .......... .......... 44% 19.5M 3s 2024-09-17T23:32:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75050K .......... .......... .......... .......... .......... 44% 37.5M 3s 2024-09-17T23:32:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75100K .......... .......... .......... .......... .......... 44% 25.9M 3s 2024-09-17T23:32:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75150K .......... .......... .......... .......... .......... 44% 23.2M 3s 2024-09-17T23:32:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75200K .......... .......... .......... .......... .......... 44% 45.1M 3s 2024-09-17T23:32:50.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75250K .......... .......... .......... .......... .......... 44% 25.3M 3s 2024-09-17T23:32:50.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75300K .......... .......... .......... .......... .......... 44% 41.8M 3s 2024-09-17T23:32:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75350K .......... .......... .......... .......... .......... 44% 22.8M 3s 2024-09-17T23:32:50.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75400K .......... .......... .......... .......... .......... 44% 23.3M 3s 2024-09-17T23:32:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75450K .......... .......... .......... .......... .......... 44% 55.3M 3s 2024-09-17T23:32:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75500K .......... .......... .......... .......... .......... 44% 29.9M 3s 2024-09-17T23:32:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75550K .......... .......... .......... .......... .......... 44% 28.3M 3s 2024-09-17T23:32:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75600K .......... .......... .......... .......... .......... 44% 22.5M 3s 2024-09-17T23:32:50.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75650K .......... .......... .......... .......... .......... 44% 26.4M 3s 2024-09-17T23:32:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75700K .......... .......... .......... .......... .......... 44% 50.1M 3s 2024-09-17T23:32:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75750K .......... .......... .......... .......... .......... 44% 23.9M 3s 2024-09-17T23:32:50.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75800K .......... .......... .......... .......... .......... 44% 38.1M 3s 2024-09-17T23:32:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75850K .......... .......... .......... .......... .......... 45% 24.2M 3s 2024-09-17T23:32:50.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75900K .......... .......... .......... .......... .......... 45% 22.0M 3s 2024-09-17T23:32:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 75950K .......... .......... .......... .......... .......... 45% 24.9M 3s 2024-09-17T23:32:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76000K .......... .......... .......... .......... .......... 45% 53.3M 3s 2024-09-17T23:32:50.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76050K .......... .......... .......... .......... .......... 45% 24.1M 3s 2024-09-17T23:32:50.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76100K .......... .......... .......... .......... .......... 45% 39.7M 3s 2024-09-17T23:32:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76150K .......... .......... .......... .......... .......... 45% 22.8M 3s 2024-09-17T23:32:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76200K .......... .......... .......... .......... .......... 45% 50.4M 3s 2024-09-17T23:32:50.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76250K .......... .......... .......... .......... .......... 45% 22.0M 3s 2024-09-17T23:32:50.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76300K .......... .......... .......... .......... .......... 45% 43.0M 3s 2024-09-17T23:32:50.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76350K .......... .......... .......... .......... .......... 45% 23.0M 3s 2024-09-17T23:32:50.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76400K .......... .......... .......... .......... .......... 45% 23.9M 3s 2024-09-17T23:32:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76450K .......... .......... .......... .......... .......... 45% 50.5M 3s 2024-09-17T23:32:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76500K .......... .......... .......... .......... .......... 45% 25.3M 3s 2024-09-17T23:32:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76550K .......... .......... .......... .......... .......... 45% 36.2M 3s 2024-09-17T23:32:50.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76600K .......... .......... .......... .......... .......... 45% 22.1M 3s 2024-09-17T23:32:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76650K .......... .......... .......... .......... .......... 45% 25.6M 3s 2024-09-17T23:32:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76700K .......... .......... .......... .......... .......... 45% 58.2M 3s 2024-09-17T23:32:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76750K .......... .......... .......... .......... .......... 45% 23.5M 3s 2024-09-17T23:32:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76800K .......... .......... .......... .......... .......... 45% 24.4M 3s 2024-09-17T23:32:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76850K .......... .......... .......... .......... .......... 45% 38.7M 3s 2024-09-17T23:32:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76900K .......... .......... .......... .......... .......... 45% 23.4M 3s 2024-09-17T23:32:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 76950K .......... .......... .......... .......... .......... 45% 59.0M 3s 2024-09-17T23:32:50.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77000K .......... .......... .......... .......... .......... 45% 29.1M 3s 2024-09-17T23:32:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77050K .......... .......... .......... .......... .......... 45% 27.9M 3s 2024-09-17T23:32:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77100K .......... .......... .......... .......... .......... 45% 23.8M 3s 2024-09-17T23:32:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77150K .......... .......... .......... .......... .......... 45% 24.0M 3s 2024-09-17T23:32:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77200K .......... .......... .......... .......... .......... 45% 24.2M 3s 2024-09-17T23:32:50.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77250K .......... .......... .......... .......... .......... 45% 53.2M 3s 2024-09-17T23:32:50.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77300K .......... .......... .......... .......... .......... 45% 23.6M 3s 2024-09-17T23:32:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77350K .......... .......... .......... .......... .......... 45% 33.5M 3s 2024-09-17T23:32:50.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77400K .......... .......... .......... .......... .......... 45% 24.8M 3s 2024-09-17T23:32:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77450K .......... .......... .......... .......... .......... 45% 52.1M 3s 2024-09-17T23:32:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77500K .......... .......... .......... .......... .......... 46% 23.8M 3s 2024-09-17T23:32:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77550K .......... .......... .......... .......... .......... 46% 24.1M 3s 2024-09-17T23:32:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77600K .......... .......... .......... .......... .......... 46% 38.8M 3s 2024-09-17T23:32:50.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77650K .......... .......... .......... .......... .......... 46% 23.7M 3s 2024-09-17T23:32:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77700K .......... .......... .......... .......... .......... 46% 23.3M 3s 2024-09-17T23:32:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77750K .......... .......... .......... .......... .......... 46% 52.8M 3s 2024-09-17T23:32:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77800K .......... .......... .......... .......... .......... 46% 23.0M 3s 2024-09-17T23:32:50.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77850K .......... .......... .......... .......... .......... 46% 42.9M 3s 2024-09-17T23:32:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77900K .......... .......... .......... .......... .......... 46% 23.4M 3s 2024-09-17T23:32:50.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 77950K .......... .......... .......... .......... .......... 46% 22.6M 3s 2024-09-17T23:32:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78000K .......... .......... .......... .......... .......... 46% 60.4M 3s 2024-09-17T23:32:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78050K .......... .......... .......... .......... .......... 46% 23.5M 3s 2024-09-17T23:32:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78100K .......... .......... .......... .......... .......... 46% 37.1M 3s 2024-09-17T23:32:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78150K .......... .......... .......... .......... .......... 46% 23.4M 3s 2024-09-17T23:32:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78200K .......... .......... .......... .......... .......... 46% 23.8M 3s 2024-09-17T23:32:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78250K .......... .......... .......... .......... .......... 46% 48.8M 3s 2024-09-17T23:32:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78300K .......... .......... .......... .......... .......... 46% 25.1M 3s 2024-09-17T23:32:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78350K .......... .......... .......... .......... .......... 46% 23.6M 3s 2024-09-17T23:32:50.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78400K .......... .......... .......... .......... .......... 46% 38.2M 3s 2024-09-17T23:32:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78450K .......... .......... .......... .......... .......... 46% 23.8M 3s 2024-09-17T23:32:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78500K .......... .......... .......... .......... .......... 46% 51.5M 3s 2024-09-17T23:32:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78550K .......... .......... .......... .......... .......... 46% 24.3M 3s 2024-09-17T23:32:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78600K .......... .......... .......... .......... .......... 46% 38.0M 3s 2024-09-17T23:32:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78650K .......... .......... .......... .......... .......... 46% 23.3M 3s 2024-09-17T23:32:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78700K .......... .......... .......... .......... .......... 46% 58.9M 3s 2024-09-17T23:32:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78750K .......... .......... .......... .......... .......... 46% 14.6M 3s 2024-09-17T23:32:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78800K .......... .......... .......... .......... .......... 46% 59.5M 3s 2024-09-17T23:32:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78850K .......... .......... .......... .......... .......... 46% 34.9M 3s 2024-09-17T23:32:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78900K .......... .......... .......... .......... .......... 46% 24.1M 3s 2024-09-17T23:32:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 78950K .......... .......... .......... .......... .......... 46% 22.2M 3s 2024-09-17T23:32:50.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79000K .......... .......... .......... .......... .......... 46% 56.2M 3s 2024-09-17T23:32:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79050K .......... .......... .......... .......... .......... 46% 24.9M 3s 2024-09-17T23:32:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79100K .......... .......... .......... .......... .......... 46% 32.2M 3s 2024-09-17T23:32:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79150K .......... .......... .......... .......... .......... 46% 26.4M 3s 2024-09-17T23:32:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79200K .......... .......... .......... .......... .......... 47% 24.3M 3s 2024-09-17T23:32:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79250K .......... .......... .......... .......... .......... 47% 52.7M 3s 2024-09-17T23:32:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79300K .......... .......... .......... .......... .......... 47% 23.7M 3s 2024-09-17T23:32:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79350K .......... .......... .......... .......... .......... 47% 36.9M 3s 2024-09-17T23:32:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79400K .......... .......... .......... .......... .......... 47% 22.3M 3s 2024-09-17T23:32:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79450K .......... .......... .......... .......... .......... 47% 25.5M 3s 2024-09-17T23:32:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79500K .......... .......... .......... .......... .......... 47% 54.3M 3s 2024-09-17T23:32:50.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79550K .......... .......... .......... .......... .......... 47% 23.8M 3s 2024-09-17T23:32:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79600K .......... .......... .......... .......... .......... 47% 24.2M 3s 2024-09-17T23:32:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79650K .......... .......... .......... .......... .......... 47% 40.1M 3s 2024-09-17T23:32:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79700K .......... .......... .......... .......... .......... 47% 23.3M 3s 2024-09-17T23:32:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79750K .......... .......... .......... .......... .......... 47% 44.8M 3s 2024-09-17T23:32:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79800K .......... .......... .......... .......... .......... 47% 24.1M 3s 2024-09-17T23:32:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79850K .......... .......... .......... .......... .......... 47% 42.3M 3s 2024-09-17T23:32:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79900K .......... .......... .......... .......... .......... 47% 22.5M 3s 2024-09-17T23:32:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 79950K .......... .......... .......... .......... .......... 47% 25.5M 3s 2024-09-17T23:32:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80000K .......... .......... .......... .......... .......... 47% 22.2M 3s 2024-09-17T23:32:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80050K .......... .......... .......... .......... .......... 47% 51.7M 3s 2024-09-17T23:32:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80100K .......... .......... .......... .......... .......... 47% 24.4M 3s 2024-09-17T23:32:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80150K .......... .......... .......... .......... .......... 47% 49.8M 3s 2024-09-17T23:32:50.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80200K .......... .......... .......... .......... .......... 47% 23.9M 3s 2024-09-17T23:32:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80250K .......... .......... .......... .......... .......... 47% 34.6M 3s 2024-09-17T23:32:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80300K .......... .......... .......... .......... .......... 47% 25.0M 3s 2024-09-17T23:32:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80350K .......... .......... .......... .......... .......... 47% 33.6M 3s 2024-09-17T23:32:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80400K .......... .......... .......... .......... .......... 47% 17.8M 3s 2024-09-17T23:32:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80450K .......... .......... .......... .......... .......... 47% 42.7M 3s 2024-09-17T23:32:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80500K .......... .......... .......... .......... .......... 47% 22.7M 3s 2024-09-17T23:32:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80550K .......... .......... .......... .......... .......... 47% 59.4M 3s 2024-09-17T23:32:50.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80600K .......... .......... .......... .......... .......... 47% 23.2M 3s 2024-09-17T23:32:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80650K .......... .......... .......... .......... .......... 47% 34.5M 3s 2024-09-17T23:32:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80700K .......... .......... .......... .......... .......... 47% 27.1M 3s 2024-09-17T23:32:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80750K .......... .......... .......... .......... .......... 47% 30.6M 3s 2024-09-17T23:32:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80800K .......... .......... .......... .......... .......... 47% 32.5M 3s 2024-09-17T23:32:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80850K .......... .......... .......... .......... .......... 47% 23.3M 3s 2024-09-17T23:32:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80900K .......... .......... .......... .......... .......... 48% 30.7M 3s 2024-09-17T23:32:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 80950K .......... .......... .......... .......... .......... 48% 36.8M 3s 2024-09-17T23:32:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81000K .......... .......... .......... .......... .......... 48% 26.1M 3s 2024-09-17T23:32:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81050K .......... .......... .......... .......... .......... 48% 41.3M 3s 2024-09-17T23:32:50.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81100K .......... .......... .......... .......... .......... 48% 21.6M 3s 2024-09-17T23:32:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81150K .......... .......... .......... .......... .......... 48% 22.3M 3s 2024-09-17T23:32:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81200K .......... .......... .......... .......... .......... 48% 38.7M 3s 2024-09-17T23:32:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81250K .......... .......... .......... .......... .......... 48% 23.9M 3s 2024-09-17T23:32:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81300K .......... .......... .......... .......... .......... 48% 53.7M 3s 2024-09-17T23:32:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81350K .......... .......... .......... .......... .......... 48% 32.9M 3s 2024-09-17T23:32:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81400K .......... .......... .......... .......... .......... 48% 27.4M 3s 2024-09-17T23:32:50.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81450K .......... .......... .......... .......... .......... 48% 23.3M 3s 2024-09-17T23:32:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81500K .......... .......... .......... .......... .......... 48% 48.3M 3s 2024-09-17T23:32:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81550K .......... .......... .......... .......... .......... 48% 20.9M 3s 2024-09-17T23:32:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81600K .......... .......... .......... .......... .......... 48% 31.9M 3s 2024-09-17T23:32:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81650K .......... .......... .......... .......... .......... 48% 20.8M 3s 2024-09-17T23:32:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81700K .......... .......... .......... .......... .......... 48% 32.3M 3s 2024-09-17T23:32:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81750K .......... .......... .......... .......... .......... 48% 26.2M 3s 2024-09-17T23:32:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81800K .......... .......... .......... .......... .......... 48% 59.1M 3s 2024-09-17T23:32:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81850K .......... .......... .......... .......... .......... 48% 23.6M 3s 2024-09-17T23:32:50.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81900K .......... .......... .......... .......... .......... 48% 34.9M 3s 2024-09-17T23:32:50.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 81950K .......... .......... .......... .......... .......... 48% 25.9M 3s 2024-09-17T23:32:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82000K .......... .......... .......... .......... .......... 48% 23.1M 3s 2024-09-17T23:32:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82050K .......... .......... .......... .......... .......... 48% 64.7M 3s 2024-09-17T23:32:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82100K .......... .......... .......... .......... .......... 48% 22.0M 3s 2024-09-17T23:32:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82150K .......... .......... .......... .......... .......... 48% 42.7M 3s 2024-09-17T23:32:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82200K .......... .......... .......... .......... .......... 48% 22.1M 3s 2024-09-17T23:32:50.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82250K .......... .......... .......... .......... .......... 48% 25.0M 3s 2024-09-17T23:32:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82300K .......... .......... .......... .......... .......... 48% 50.0M 3s 2024-09-17T23:32:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82350K .......... .......... .......... .......... .......... 48% 23.1M 3s 2024-09-17T23:32:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82400K .......... .......... .......... .......... .......... 48% 24.5M 3s 2024-09-17T23:32:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82450K .......... .......... .......... .......... .......... 48% 39.2M 3s 2024-09-17T23:32:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82500K .......... .......... .......... .......... .......... 48% 28.4M 3s 2024-09-17T23:32:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82550K .......... .......... .......... .......... .......... 49% 33.7M 3s 2024-09-17T23:32:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82600K .......... .......... .......... .......... .......... 49% 24.8M 3s 2024-09-17T23:32:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82650K .......... .......... .......... .......... .......... 49% 42.0M 3s 2024-09-17T23:32:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82700K .......... .......... .......... .......... .......... 49% 22.8M 3s 2024-09-17T23:32:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82750K .......... .......... .......... .......... .......... 49% 20.0M 3s 2024-09-17T23:32:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82800K .......... .......... .......... .......... .......... 49% 29.3M 3s 2024-09-17T23:32:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82850K .......... .......... .......... .......... .......... 49% 54.9M 3s 2024-09-17T23:32:50.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82900K .......... .......... .......... .......... .......... 49% 23.7M 3s 2024-09-17T23:32:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 82950K .......... .......... .......... .......... .......... 49% 37.2M 3s 2024-09-17T23:32:50.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83000K .......... .......... .......... .......... .......... 49% 25.8M 3s 2024-09-17T23:32:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83050K .......... .......... .......... .......... .......... 49% 49.0M 3s 2024-09-17T23:32:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83100K .......... .......... .......... .......... .......... 49% 23.3M 3s 2024-09-17T23:32:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83150K .......... .......... .......... .......... .......... 49% 23.3M 3s 2024-09-17T23:32:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83200K .......... .......... .......... .......... .......... 49% 37.4M 3s 2024-09-17T23:32:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83250K .......... .......... .......... .......... .......... 49% 25.2M 3s 2024-09-17T23:32:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83300K .......... .......... .......... .......... .......... 49% 26.3M 3s 2024-09-17T23:32:50.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83350K .......... .......... .......... .......... .......... 49% 37.2M 3s 2024-09-17T23:32:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83400K .......... .......... .......... .......... .......... 49% 24.2M 3s 2024-09-17T23:32:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83450K .......... .......... .......... .......... .......... 49% 55.5M 3s 2024-09-17T23:32:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83500K .......... .......... .......... .......... .......... 49% 21.6M 3s 2024-09-17T23:32:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83550K .......... .......... .......... .......... .......... 49% 24.3M 3s 2024-09-17T23:32:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83600K .......... .......... .......... .......... .......... 49% 47.7M 3s 2024-09-17T23:32:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83650K .......... .......... .......... .......... .......... 49% 23.3M 3s 2024-09-17T23:32:50.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83700K .......... .......... .......... .......... .......... 49% 41.7M 3s 2024-09-17T23:32:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83750K .......... .......... .......... .......... .......... 49% 23.2M 3s 2024-09-17T23:32:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83800K .......... .......... .......... .......... .......... 49% 52.8M 3s 2024-09-17T23:32:50.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83850K .......... .......... .......... .......... .......... 49% 24.0M 3s 2024-09-17T23:32:50.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83900K .......... .......... .......... .......... .......... 49% 32.1M 3s 2024-09-17T23:32:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 83950K .......... .......... .......... .......... .......... 49% 25.7M 3s 2024-09-17T23:32:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84000K .......... .......... .......... .......... .......... 49% 22.6M 3s 2024-09-17T23:32:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84050K .......... .......... .......... .......... .......... 49% 25.4M 3s 2024-09-17T23:32:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84100K .......... .......... .......... .......... .......... 49% 54.5M 3s 2024-09-17T23:32:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84150K .......... .......... .......... .......... .......... 49% 23.3M 3s 2024-09-17T23:32:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84200K .......... .......... .......... .......... .......... 49% 28.0M 3s 2024-09-17T23:32:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84250K .......... .......... .......... .......... .......... 50% 31.3M 3s 2024-09-17T23:32:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84300K .......... .......... .......... .......... .......... 50% 55.8M 3s 2024-09-17T23:32:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84350K .......... .......... .......... .......... .......... 50% 22.7M 3s 2024-09-17T23:32:50.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84400K .......... .......... .......... .......... .......... 50% 23.5M 3s 2024-09-17T23:32:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84450K .......... .......... .......... .......... .......... 50% 38.2M 3s 2024-09-17T23:32:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84500K .......... .......... .......... .......... .......... 50% 23.3M 3s 2024-09-17T23:32:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84550K .......... .......... .......... .......... .......... 50% 23.4M 3s 2024-09-17T23:32:50.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84600K .......... .......... .......... .......... .......... 50% 61.8M 3s 2024-09-17T23:32:50.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84650K .......... .......... .......... .......... .......... 50% 24.1M 3s 2024-09-17T23:32:50.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84700K .......... .......... .......... .......... .......... 50% 38.7M 3s 2024-09-17T23:32:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84750K .......... .......... .......... .......... .......... 50% 23.3M 3s 2024-09-17T23:32:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84800K .......... .......... .......... .......... .......... 50% 23.8M 3s 2024-09-17T23:32:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84850K .......... .......... .......... .......... .......... 50% 53.3M 3s 2024-09-17T23:32:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84900K .......... .......... .......... .......... .......... 50% 23.6M 3s 2024-09-17T23:32:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 84950K .......... .......... .......... .......... .......... 50% 36.4M 3s 2024-09-17T23:32:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85000K .......... .......... .......... .......... .......... 50% 22.4M 3s 2024-09-17T23:32:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85050K .......... .......... .......... .......... .......... 50% 24.4M 3s 2024-09-17T23:32:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85100K .......... .......... .......... .......... .......... 50% 62.5M 3s 2024-09-17T23:32:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85150K .......... .......... .......... .......... .......... 50% 23.5M 3s 2024-09-17T23:32:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85200K .......... .......... .......... .......... .......... 50% 23.0M 3s 2024-09-17T23:32:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85250K .......... .......... .......... .......... .......... 50% 14.2M 3s 2024-09-17T23:32:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85300K .......... .......... .......... .......... .......... 50% 116M 3s 2024-09-17T23:32:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85350K .......... .......... .......... .......... .......... 50% 186M 3s 2024-09-17T23:32:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85400K .......... .......... .......... .......... .......... 50% 23.0M 3s 2024-09-17T23:32:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85450K .......... .......... .......... .......... .......... 50% 35.5M 3s 2024-09-17T23:32:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85500K .......... .......... .......... .......... .......... 50% 26.1M 3s 2024-09-17T23:32:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85550K .......... .......... .......... .......... .......... 50% 22.9M 3s 2024-09-17T23:32:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85600K .......... .......... .......... .......... .......... 50% 28.9M 3s 2024-09-17T23:32:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85650K .......... .......... .......... .......... .......... 50% 39.3M 3s 2024-09-17T23:32:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85700K .......... .......... .......... .......... .......... 50% 23.0M 3s 2024-09-17T23:32:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85750K .......... .......... .......... .......... .......... 50% 37.7M 3s 2024-09-17T23:32:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85800K .......... .......... .......... .......... .......... 50% 25.0M 3s 2024-09-17T23:32:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85850K .......... .......... .......... .......... .......... 50% 51.7M 3s 2024-09-17T23:32:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85900K .......... .......... .......... .......... .......... 50% 22.0M 3s 2024-09-17T23:32:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 85950K .......... .......... .......... .......... .......... 51% 25.2M 3s 2024-09-17T23:32:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86000K .......... .......... .......... .......... .......... 51% 28.6M 3s 2024-09-17T23:32:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86050K .......... .......... .......... .......... .......... 51% 29.4M 3s 2024-09-17T23:32:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86100K .......... .......... .......... .......... .......... 51% 23.8M 3s 2024-09-17T23:32:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86150K .......... .......... .......... .......... .......... 51% 54.3M 3s 2024-09-17T23:32:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86200K .......... .......... .......... .......... .......... 51% 35.4M 3s 2024-09-17T23:32:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86250K .......... .......... .......... .......... .......... 51% 25.2M 3s 2024-09-17T23:32:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86300K .......... .......... .......... .......... .......... 51% 24.2M 3s 2024-09-17T23:32:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86350K .......... .......... .......... .......... .......... 51% 23.6M 3s 2024-09-17T23:32:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86400K .......... .......... .......... .......... .......... 51% 48.1M 3s 2024-09-17T23:32:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86450K .......... .......... .......... .......... .......... 51% 24.7M 3s 2024-09-17T23:32:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86500K .......... .......... .......... .......... .......... 51% 40.2M 3s 2024-09-17T23:32:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86550K .......... .......... .......... .......... .......... 51% 22.5M 3s 2024-09-17T23:32:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86600K .......... .......... .......... .......... .......... 51% 51.6M 3s 2024-09-17T23:32:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86650K .......... .......... .......... .......... .......... 51% 24.0M 3s 2024-09-17T23:32:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86700K .......... .......... .......... .......... .......... 51% 34.6M 3s 2024-09-17T23:32:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86750K .......... .......... .......... .......... .......... 51% 25.2M 3s 2024-09-17T23:32:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86800K .......... .......... .......... .......... .......... 51% 23.5M 3s 2024-09-17T23:32:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86850K .......... .......... .......... .......... .......... 51% 24.4M 3s 2024-09-17T23:32:50.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86900K .......... .......... .......... .......... .......... 51% 55.0M 3s 2024-09-17T23:32:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 86950K .......... .......... .......... .......... .......... 51% 23.5M 3s 2024-09-17T23:32:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87000K .......... .......... .......... .......... .......... 51% 40.5M 3s 2024-09-17T23:32:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87050K .......... .......... .......... .......... .......... 51% 24.9M 3s 2024-09-17T23:32:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87100K .......... .......... .......... .......... .......... 51% 48.0M 3s 2024-09-17T23:32:50.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87150K .......... .......... .......... .......... .......... 51% 23.7M 3s 2024-09-17T23:32:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87200K .......... .......... .......... .......... .......... 51% 23.9M 3s 2024-09-17T23:32:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87250K .......... .......... .......... .......... .......... 51% 37.8M 3s 2024-09-17T23:32:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87300K .......... .......... .......... .......... .......... 51% 23.9M 3s 2024-09-17T23:32:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87350K .......... .......... .......... .......... .......... 51% 22.2M 3s 2024-09-17T23:32:50.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87400K .......... .......... .......... .......... .......... 51% 58.2M 3s 2024-09-17T23:32:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87450K .......... .......... .......... .......... .......... 51% 24.3M 3s 2024-09-17T23:32:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87500K .......... .......... .......... .......... .......... 51% 38.5M 3s 2024-09-17T23:32:50.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87550K .......... .......... .......... .......... .......... 51% 22.0M 3s 2024-09-17T23:32:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87600K .......... .......... .......... .......... .......... 51% 25.4M 3s 2024-09-17T23:32:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87650K .......... .......... .......... .......... .......... 52% 60.5M 3s 2024-09-17T23:32:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87700K .......... .......... .......... .......... .......... 52% 23.0M 3s 2024-09-17T23:32:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87750K .......... .......... .......... .......... .......... 52% 23.3M 3s 2024-09-17T23:32:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87800K .......... .......... .......... .......... .......... 52% 36.9M 3s 2024-09-17T23:32:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87850K .......... .......... .......... .......... .......... 52% 25.3M 3s 2024-09-17T23:32:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87900K .......... .......... .......... .......... .......... 52% 70.7M 3s 2024-09-17T23:32:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 87950K .......... .......... .......... .......... .......... 52% 20.5M 3s 2024-09-17T23:32:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88000K .......... .......... .......... .......... .......... 52% 23.5M 3s 2024-09-17T23:32:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88050K .......... .......... .......... .......... .......... 52% 39.2M 3s 2024-09-17T23:32:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88100K .......... .......... .......... .......... .......... 52% 23.7M 3s 2024-09-17T23:32:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88150K .......... .......... .......... .......... .......... 52% 52.2M 3s 2024-09-17T23:32:50.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88200K .......... .......... .......... .......... .......... 52% 23.1M 3s 2024-09-17T23:32:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88250K .......... .......... .......... .......... .......... 52% 40.0M 3s 2024-09-17T23:32:50.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88300K .......... .......... .......... .......... .......... 52% 21.7M 3s 2024-09-17T23:32:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88350K .......... .......... .......... .......... .......... 52% 26.6M 3s 2024-09-17T23:32:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88400K .......... .......... .......... .......... .......... 52% 23.0M 3s 2024-09-17T23:32:50.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88450K .......... .......... .......... .......... .......... 52% 57.9M 3s 2024-09-17T23:32:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88500K .......... .......... .......... .......... .......... 52% 32.6M 3s 2024-09-17T23:32:50.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88550K .......... .......... .......... .......... .......... 52% 25.3M 3s 2024-09-17T23:32:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88600K .......... .......... .......... .......... .......... 52% 24.7M 3s 2024-09-17T23:32:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88650K .......... .......... .......... .......... .......... 52% 53.1M 3s 2024-09-17T23:32:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88700K .......... .......... .......... .......... .......... 52% 24.9M 3s 2024-09-17T23:32:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88750K .......... .......... .......... .......... .......... 52% 22.1M 3s 2024-09-17T23:32:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88800K .......... .......... .......... .......... .......... 52% 41.6M 3s 2024-09-17T23:32:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88850K .......... .......... .......... .......... .......... 52% 23.7M 3s 2024-09-17T23:32:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88900K .......... .......... .......... .......... .......... 52% 52.7M 3s 2024-09-17T23:32:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 88950K .......... .......... .......... .......... .......... 52% 23.6M 3s 2024-09-17T23:32:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89000K .......... .......... .......... .......... .......... 52% 39.3M 3s 2024-09-17T23:32:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89050K .......... .......... .......... .......... .......... 52% 23.5M 3s 2024-09-17T23:32:50.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89100K .......... .......... .......... .......... .......... 52% 22.5M 3s 2024-09-17T23:32:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89150K .......... .......... .......... .......... .......... 52% 28.3M 3s 2024-09-17T23:32:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89200K .......... .......... .......... .......... .......... 52% 39.6M 3s 2024-09-17T23:32:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89250K .......... .......... .......... .......... .......... 52% 25.0M 3s 2024-09-17T23:32:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89300K .......... .......... .......... .......... .......... 53% 36.1M 3s 2024-09-17T23:32:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89350K .......... .......... .......... .......... .......... 53% 23.4M 3s 2024-09-17T23:32:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89400K .......... .......... .......... .......... .......... 53% 61.0M 3s 2024-09-17T23:32:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89450K .......... .......... .......... .......... .......... 53% 23.8M 3s 2024-09-17T23:32:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89500K .......... .......... .......... .......... .......... 53% 33.1M 3s 2024-09-17T23:32:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89550K .......... .......... .......... .......... .......... 53% 26.0M 3s 2024-09-17T23:32:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89600K .......... .......... .......... .......... .......... 53% 22.5M 3s 2024-09-17T23:32:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89650K .......... .......... .......... .......... .......... 53% 24.6M 3s 2024-09-17T23:32:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89700K .......... .......... .......... .......... .......... 53% 43.2M 3s 2024-09-17T23:32:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89750K .......... .......... .......... .......... .......... 53% 28.9M 3s 2024-09-17T23:32:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89800K .......... .......... .......... .......... .......... 53% 32.4M 3s 2024-09-17T23:32:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89850K .......... .......... .......... .......... .......... 53% 23.3M 3s 2024-09-17T23:32:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89900K .......... .......... .......... .......... .......... 53% 57.9M 3s 2024-09-17T23:32:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 89950K .......... .......... .......... .......... .......... 53% 24.0M 3s 2024-09-17T23:32:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90000K .......... .......... .......... .......... .......... 53% 22.9M 3s 2024-09-17T23:32:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90050K .......... .......... .......... .......... .......... 53% 40.3M 3s 2024-09-17T23:32:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90100K .......... .......... .......... .......... .......... 53% 21.3M 3s 2024-09-17T23:32:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90150K .......... .......... .......... .......... .......... 53% 24.6M 3s 2024-09-17T23:32:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90200K .......... .......... .......... .......... .......... 53% 66.2M 3s 2024-09-17T23:32:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90250K .......... .......... .......... .......... .......... 53% 23.0M 3s 2024-09-17T23:32:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90300K .......... .......... .......... .......... .......... 53% 41.5M 3s 2024-09-17T23:32:50.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90350K .......... .......... .......... .......... .......... 53% 21.9M 3s 2024-09-17T23:32:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90400K .......... .......... .......... .......... .......... 53% 24.9M 3s 2024-09-17T23:32:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90450K .......... .......... .......... .......... .......... 53% 50.2M 3s 2024-09-17T23:32:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90500K .......... .......... .......... .......... .......... 53% 24.1M 3s 2024-09-17T23:32:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90550K .......... .......... .......... .......... .......... 53% 34.9M 3s 2024-09-17T23:32:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90600K .......... .......... .......... .......... .......... 53% 24.7M 3s 2024-09-17T23:32:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90650K .......... .......... .......... .......... .......... 53% 24.1M 3s 2024-09-17T23:32:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90700K .......... .......... .......... .......... .......... 53% 50.8M 3s 2024-09-17T23:32:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90750K .......... .......... .......... .......... .......... 53% 24.0M 3s 2024-09-17T23:32:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90800K .......... .......... .......... .......... .......... 53% 23.9M 3s 2024-09-17T23:32:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90850K .......... .......... .......... .......... .......... 53% 35.5M 3s 2024-09-17T23:32:50.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90900K .......... .......... .......... .......... .......... 53% 25.9M 3s 2024-09-17T23:32:50.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 90950K .......... .......... .......... .......... .......... 53% 50.0M 3s 2024-09-17T23:32:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91000K .......... .......... .......... .......... .......... 54% 22.8M 3s 2024-09-17T23:32:50.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91050K .......... .......... .......... .......... .......... 54% 43.9M 3s 2024-09-17T23:32:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91100K .......... .......... .......... .......... .......... 54% 22.7M 3s 2024-09-17T23:32:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91150K .......... .......... .......... .......... .......... 54% 23.5M 3s 2024-09-17T23:32:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91200K .......... .......... .......... .......... .......... 54% 22.8M 3s 2024-09-17T23:32:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91250K .......... .......... .......... .......... .......... 54% 60.6M 3s 2024-09-17T23:32:50.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91300K .......... .......... .......... .......... .......... 54% 37.5M 3s 2024-09-17T23:32:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91350K .......... .......... .......... .......... .......... 54% 24.0M 3s 2024-09-17T23:32:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91400K .......... .......... .......... .......... .......... 54% 22.6M 3s 2024-09-17T23:32:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91450K .......... .......... .......... .......... .......... 54% 60.7M 3s 2024-09-17T23:32:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91500K .......... .......... .......... .......... .......... 54% 23.7M 3s 2024-09-17T23:32:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91550K .......... .......... .......... .......... .......... 54% 23.5M 3s 2024-09-17T23:32:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91600K .......... .......... .......... .......... .......... 54% 41.6M 3s 2024-09-17T23:32:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91650K .......... .......... .......... .......... .......... 54% 23.5M 3s 2024-09-17T23:32:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91700K .......... .......... .......... .......... .......... 54% 52.5M 3s 2024-09-17T23:32:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91750K .......... .......... .......... .......... .......... 54% 24.8M 3s 2024-09-17T23:32:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91800K .......... .......... .......... .......... .......... 54% 31.7M 3s 2024-09-17T23:32:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91850K .......... .......... .......... .......... .......... 54% 25.3M 3s 2024-09-17T23:32:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91900K .......... .......... .......... .......... .......... 54% 24.1M 3s 2024-09-17T23:32:50.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 91950K .......... .......... .......... .......... .......... 54% 23.8M 3s 2024-09-17T23:32:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92000K .......... .......... .......... .......... .......... 54% 46.2M 3s 2024-09-17T23:32:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92050K .......... .......... .......... .......... .......... 54% 24.6M 3s 2024-09-17T23:32:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92100K .......... .......... .......... .......... .......... 54% 39.1M 3s 2024-09-17T23:32:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92150K .......... .......... .......... .......... .......... 54% 23.4M 3s 2024-09-17T23:32:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92200K .......... .......... .......... .......... .......... 54% 55.3M 3s 2024-09-17T23:32:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92250K .......... .......... .......... .......... .......... 54% 22.8M 3s 2024-09-17T23:32:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92300K .......... .......... .......... .......... .......... 54% 44.0M 3s 2024-09-17T23:32:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92350K .......... .......... .......... .......... .......... 54% 23.3M 3s 2024-09-17T23:32:50.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92400K .......... .......... .......... .......... .......... 54% 21.8M 3s 2024-09-17T23:32:50.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92450K .......... .......... .......... .......... .......... 54% 24.6M 3s 2024-09-17T23:32:50.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92500K .......... .......... .......... .......... .......... 54% 54.4M 3s 2024-09-17T23:32:50.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92550K .......... .......... .......... .......... .......... 54% 23.2M 3s 2024-09-17T23:32:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92600K .......... .......... .......... .......... .......... 54% 43.4M 3s 2024-09-17T23:32:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92650K .......... .......... .......... .......... .......... 54% 23.4M 3s 2024-09-17T23:32:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92700K .......... .......... .......... .......... .......... 55% 53.0M 3s 2024-09-17T23:32:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92750K .......... .......... .......... .......... .......... 55% 23.1M 3s 2024-09-17T23:32:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92800K .......... .......... .......... .......... .......... 55% 25.6M 3s 2024-09-17T23:32:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92850K .......... .......... .......... .......... .......... 55% 33.2M 3s 2024-09-17T23:32:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92900K .......... .......... .......... .......... .......... 55% 23.7M 3s 2024-09-17T23:32:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 92950K .......... .......... .......... .......... .......... 55% 25.0M 3s 2024-09-17T23:32:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93000K .......... .......... .......... .......... .......... 55% 54.4M 3s 2024-09-17T23:32:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93050K .......... .......... .......... .......... .......... 55% 23.5M 3s 2024-09-17T23:32:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93100K .......... .......... .......... .......... .......... 55% 38.7M 3s 2024-09-17T23:32:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93150K .......... .......... .......... .......... .......... 55% 23.2M 3s 2024-09-17T23:32:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93200K .......... .......... .......... .......... .......... 55% 23.4M 3s 2024-09-17T23:32:50.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93250K .......... .......... .......... .......... .......... 55% 50.9M 3s 2024-09-17T23:32:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93300K .......... .......... .......... .......... .......... 55% 25.2M 3s 2024-09-17T23:32:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93350K .......... .......... .......... .......... .......... 55% 36.0M 3s 2024-09-17T23:32:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93400K .......... .......... .......... .......... .......... 55% 27.3M 3s 2024-09-17T23:32:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93450K .......... .......... .......... .......... .......... 55% 25.3M 3s 2024-09-17T23:32:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93500K .......... .......... .......... .......... .......... 55% 34.2M 3s 2024-09-17T23:32:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93550K .......... .......... .......... .......... .......... 55% 24.2M 3s 2024-09-17T23:32:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93600K .......... .......... .......... .......... .......... 55% 24.8M 3s 2024-09-17T23:32:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93650K .......... .......... .......... .......... .......... 55% 35.7M 3s 2024-09-17T23:32:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93700K .......... .......... .......... .......... .......... 55% 37.5M 3s 2024-09-17T23:32:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93750K .......... .......... .......... .......... .......... 55% 35.5M 3s 2024-09-17T23:32:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93800K .......... .......... .......... .......... .......... 55% 21.7M 3s 2024-09-17T23:32:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93850K .......... .......... .......... .......... .......... 55% 30.2M 3s 2024-09-17T23:32:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93900K .......... .......... .......... .......... .......... 55% 29.9M 3s 2024-09-17T23:32:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 93950K .......... .......... .......... .......... .......... 55% 22.9M 3s 2024-09-17T23:32:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94000K .......... .......... .......... .......... .......... 55% 46.0M 3s 2024-09-17T23:32:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94050K .......... .......... .......... .......... .......... 55% 24.4M 3s 2024-09-17T23:32:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94100K .......... .......... .......... .......... .......... 55% 28.9M 3s 2024-09-17T23:32:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94150K .......... .......... .......... .......... .......... 55% 40.2M 3s 2024-09-17T23:32:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94200K .......... .......... .......... .......... .......... 55% 22.5M 3s 2024-09-17T23:32:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94250K .......... .......... .......... .......... .......... 55% 42.4M 3s 2024-09-17T23:32:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94300K .......... .......... .......... .......... .......... 55% 32.4M 3s 2024-09-17T23:32:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94350K .......... .......... .......... .......... .......... 56% 18.6M 3s 2024-09-17T23:32:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94400K .......... .......... .......... .......... .......... 56% 43.4M 3s 2024-09-17T23:32:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94450K .......... .......... .......... .......... .......... 56% 25.4M 3s 2024-09-17T23:32:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94500K .......... .......... .......... .......... .......... 56% 38.8M 3s 2024-09-17T23:32:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94550K .......... .......... .......... .......... .......... 56% 24.9M 3s 2024-09-17T23:32:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94600K .......... .......... .......... .......... .......... 56% 36.3M 3s 2024-09-17T23:32:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94650K .......... .......... .......... .......... .......... 56% 25.0M 3s 2024-09-17T23:32:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94700K .......... .......... .......... .......... .......... 56% 29.3M 3s 2024-09-17T23:32:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94750K .......... .......... .......... .......... .......... 56% 18.8M 3s 2024-09-17T23:32:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94800K .......... .......... .......... .......... .......... 56% 50.6M 3s 2024-09-17T23:32:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94850K .......... .......... .......... .......... .......... 56% 26.7M 3s 2024-09-17T23:32:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94900K .......... .......... .......... .......... .......... 56% 34.2M 3s 2024-09-17T23:32:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 94950K .......... .......... .......... .......... .......... 56% 24.0M 3s 2024-09-17T23:32:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95000K .......... .......... .......... .......... .......... 56% 52.5M 3s 2024-09-17T23:32:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95050K .......... .......... .......... .......... .......... 56% 25.2M 3s 2024-09-17T23:32:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95100K .......... .......... .......... .......... .......... 56% 37.7M 3s 2024-09-17T23:32:50.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95150K .......... .......... .......... .......... .......... 56% 23.3M 3s 2024-09-17T23:32:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95200K .......... .......... .......... .......... .......... 56% 22.6M 3s 2024-09-17T23:32:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95250K .......... .......... .......... .......... .......... 56% 24.5M 3s 2024-09-17T23:32:50.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95300K .......... .......... .......... .......... .......... 56% 60.3M 3s 2024-09-17T23:32:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95350K .......... .......... .......... .......... .......... 56% 23.0M 3s 2024-09-17T23:32:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95400K .......... .......... .......... .......... .......... 56% 40.4M 3s 2024-09-17T23:32:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95450K .......... .......... .......... .......... .......... 56% 22.7M 3s 2024-09-17T23:32:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95500K .......... .......... .......... .......... .......... 56% 55.4M 3s 2024-09-17T23:32:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95550K .......... .......... .......... .......... .......... 56% 22.9M 3s 2024-09-17T23:32:50.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95600K .......... .......... .......... .......... .......... 56% 23.7M 3s 2024-09-17T23:32:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95650K .......... .......... .......... .......... .......... 56% 39.7M 3s 2024-09-17T23:32:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95700K .......... .......... .......... .......... .......... 56% 20.3M 2s 2024-09-17T23:32:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95750K .......... .......... .......... .......... .......... 56% 26.9M 2s 2024-09-17T23:32:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95800K .......... .......... .......... .......... .......... 56% 48.7M 2s 2024-09-17T23:32:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95850K .......... .......... .......... .......... .......... 56% 37.7M 2s 2024-09-17T23:32:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95900K .......... .......... .......... .......... .......... 56% 24.1M 2s 2024-09-17T23:32:50.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 95950K .......... .......... .......... .......... .......... 56% 25.8M 2s 2024-09-17T23:32:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96000K .......... .......... .......... .......... .......... 56% 22.8M 2s 2024-09-17T23:32:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96050K .......... .......... .......... .......... .......... 57% 55.0M 2s 2024-09-17T23:32:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96100K .......... .......... .......... .......... .......... 57% 24.6M 2s 2024-09-17T23:32:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96150K .......... .......... .......... .......... .......... 57% 35.5M 2s 2024-09-17T23:32:50.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96200K .......... .......... .......... .......... .......... 57% 24.7M 2s 2024-09-17T23:32:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96250K .......... .......... .......... .......... .......... 57% 53.9M 2s 2024-09-17T23:32:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96300K .......... .......... .......... .......... .......... 57% 22.3M 2s 2024-09-17T23:32:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96350K .......... .......... .......... .......... .......... 57% 25.4M 2s 2024-09-17T23:32:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96400K .......... .......... .......... .......... .......... 57% 37.0M 2s 2024-09-17T23:32:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96450K .......... .......... .......... .......... .......... 57% 23.1M 2s 2024-09-17T23:32:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96500K .......... .......... .......... .......... .......... 57% 25.8M 2s 2024-09-17T23:32:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96550K .......... .......... .......... .......... .......... 57% 42.0M 2s 2024-09-17T23:32:50.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96600K .......... .......... .......... .......... .......... 57% 25.0M 2s 2024-09-17T23:32:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96650K .......... .......... .......... .......... .......... 57% 45.5M 2s 2024-09-17T23:32:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96700K .......... .......... .......... .......... .......... 57% 23.5M 2s 2024-09-17T23:32:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96750K .......... .......... .......... .......... .......... 57% 23.7M 2s 2024-09-17T23:32:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96800K .......... .......... .......... .......... .......... 57% 47.6M 2s 2024-09-17T23:32:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96850K .......... .......... .......... .......... .......... 57% 23.7M 2s 2024-09-17T23:32:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96900K .......... .......... .......... .......... .......... 57% 36.3M 2s 2024-09-17T23:32:50.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 96950K .......... .......... .......... .......... .......... 57% 23.4M 2s 2024-09-17T23:32:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97000K .......... .......... .......... .......... .......... 57% 24.1M 2s 2024-09-17T23:32:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97050K .......... .......... .......... .......... .......... 57% 51.5M 2s 2024-09-17T23:32:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97100K .......... .......... .......... .......... .......... 57% 24.7M 2s 2024-09-17T23:32:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97150K .......... .......... .......... .......... .......... 57% 26.5M 2s 2024-09-17T23:32:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97200K .......... .......... .......... .......... .......... 57% 33.1M 2s 2024-09-17T23:32:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97250K .......... .......... .......... .......... .......... 57% 23.8M 2s 2024-09-17T23:32:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97300K .......... .......... .......... .......... .......... 57% 54.8M 2s 2024-09-17T23:32:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97350K .......... .......... .......... .......... .......... 57% 23.5M 2s 2024-09-17T23:32:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97400K .......... .......... .......... .......... .......... 57% 32.9M 2s 2024-09-17T23:32:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97450K .......... .......... .......... .......... .......... 57% 22.7M 2s 2024-09-17T23:32:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97500K .......... .......... .......... .......... .......... 57% 24.8M 2s 2024-09-17T23:32:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97550K .......... .......... .......... .......... .......... 57% 23.5M 2s 2024-09-17T23:32:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97600K .......... .......... .......... .......... .......... 57% 53.1M 2s 2024-09-17T23:32:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97650K .......... .......... .......... .......... .......... 57% 24.5M 2s 2024-09-17T23:32:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97700K .......... .......... .......... .......... .......... 57% 39.3M 2s 2024-09-17T23:32:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97750K .......... .......... .......... .......... .......... 58% 24.3M 2s 2024-09-17T23:32:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97800K .......... .......... .......... .......... .......... 58% 47.5M 2s 2024-09-17T23:32:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97850K .......... .......... .......... .......... .......... 58% 23.4M 2s 2024-09-17T23:32:50.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97900K .......... .......... .......... .......... .......... 58% 34.9M 2s 2024-09-17T23:32:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 97950K .......... .......... .......... .......... .......... 58% 23.7M 2s 2024-09-17T23:32:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98000K .......... .......... .......... .......... .......... 58% 22.6M 2s 2024-09-17T23:32:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98050K .......... .......... .......... .......... .......... 58% 27.6M 2s 2024-09-17T23:32:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98100K .......... .......... .......... .......... .......... 58% 52.7M 2s 2024-09-17T23:32:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98150K .......... .......... .......... .......... .......... 58% 22.2M 2s 2024-09-17T23:32:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98200K .......... .......... .......... .......... .......... 58% 37.1M 2s 2024-09-17T23:32:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98250K .......... .......... .......... .......... .......... 58% 26.1M 2s 2024-09-17T23:32:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98300K .......... .......... .......... .......... .......... 58% 41.9M 2s 2024-09-17T23:32:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98350K .......... .......... .......... .......... .......... 58% 24.5M 2s 2024-09-17T23:32:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98400K .......... .......... .......... .......... .......... 58% 24.9M 2s 2024-09-17T23:32:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98450K .......... .......... .......... .......... .......... 58% 36.0M 2s 2024-09-17T23:32:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98500K .......... .......... .......... .......... .......... 58% 25.9M 2s 2024-09-17T23:32:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98550K .......... .......... .......... .......... .......... 58% 27.1M 2s 2024-09-17T23:32:50.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98600K .......... .......... .......... .......... .......... 58% 36.9M 2s 2024-09-17T23:32:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98650K .......... .......... .......... .......... .......... 58% 37.0M 2s 2024-09-17T23:32:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98700K .......... .......... .......... .......... .......... 58% 23.7M 2s 2024-09-17T23:32:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98750K .......... .......... .......... .......... .......... 58% 28.1M 2s 2024-09-17T23:32:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98800K .......... .......... .......... .......... .......... 58% 24.4M 2s 2024-09-17T23:32:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98850K .......... .......... .......... .......... .......... 58% 37.3M 2s 2024-09-17T23:32:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98900K .......... .......... .......... .......... .......... 58% 24.0M 2s 2024-09-17T23:32:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 98950K .......... .......... .......... .......... .......... 58% 57.0M 2s 2024-09-17T23:32:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99000K .......... .......... .......... .......... .......... 58% 21.5M 2s 2024-09-17T23:32:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99050K .......... .......... .......... .......... .......... 58% 47.9M 2s 2024-09-17T23:32:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99100K .......... .......... .......... .......... .......... 58% 24.1M 2s 2024-09-17T23:32:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99150K .......... .......... .......... .......... .......... 58% 23.4M 2s 2024-09-17T23:32:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99200K .......... .......... .......... .......... .......... 58% 38.2M 2s 2024-09-17T23:32:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99250K .......... .......... .......... .......... .......... 58% 22.8M 2s 2024-09-17T23:32:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99300K .......... .......... .......... .......... .......... 58% 25.3M 2s 2024-09-17T23:32:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99350K .......... .......... .......... .......... .......... 58% 49.8M 2s 2024-09-17T23:32:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99400K .......... .......... .......... .......... .......... 58% 23.4M 2s 2024-09-17T23:32:51.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99450K .......... .......... .......... .......... .......... 59% 36.7M 2s 2024-09-17T23:32:51.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99500K .......... .......... .......... .......... .......... 59% 26.1M 2s 2024-09-17T23:32:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99550K .......... .......... .......... .......... .......... 59% 23.1M 2s 2024-09-17T23:32:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99600K .......... .......... .......... .......... .......... 59% 53.7M 2s 2024-09-17T23:32:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99650K .......... .......... .......... .......... .......... 59% 23.6M 2s 2024-09-17T23:32:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99700K .......... .......... .......... .......... .......... 59% 36.1M 2s 2024-09-17T23:32:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99750K .......... .......... .......... .......... .......... 59% 24.9M 2s 2024-09-17T23:32:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99800K .......... .......... .......... .......... .......... 59% 21.8M 2s 2024-09-17T23:32:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99850K .......... .......... .......... .......... .......... 59% 57.6M 2s 2024-09-17T23:32:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99900K .......... .......... .......... .......... .......... 59% 23.2M 2s 2024-09-17T23:32:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 99950K .......... .......... .......... .......... .......... 59% 23.6M 2s 2024-09-17T23:32:51.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100000K .......... .......... .......... .......... .......... 59% 45.9M 2s 2024-09-17T23:32:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100050K .......... .......... .......... .......... .......... 59% 22.3M 2s 2024-09-17T23:32:51.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100100K .......... .......... .......... .......... .......... 59% 44.4M 2s 2024-09-17T23:32:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100150K .......... .......... .......... .......... .......... 59% 24.4M 2s 2024-09-17T23:32:51.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100200K .......... .......... .......... .......... .......... 59% 36.3M 2s 2024-09-17T23:32:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100250K .......... .......... .......... .......... .......... 59% 23.7M 2s 2024-09-17T23:32:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100300K .......... .......... .......... .......... .......... 59% 24.1M 2s 2024-09-17T23:32:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100350K .......... .......... .......... .......... .......... 59% 23.6M 2s 2024-09-17T23:32:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100400K .......... .......... .......... .......... .......... 59% 60.6M 2s 2024-09-17T23:32:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100450K .......... .......... .......... .......... .......... 59% 24.3M 2s 2024-09-17T23:32:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100500K .......... .......... .......... .......... .......... 59% 33.7M 2s 2024-09-17T23:32:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100550K .......... .......... .......... .......... .......... 59% 23.8M 2s 2024-09-17T23:32:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100600K .......... .......... .......... .......... .......... 59% 47.7M 2s 2024-09-17T23:32:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100650K .......... .......... .......... .......... .......... 59% 25.1M 2s 2024-09-17T23:32:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100700K .......... .......... .......... .......... .......... 59% 37.7M 2s 2024-09-17T23:32:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100750K .......... .......... .......... .......... .......... 59% 22.7M 2s 2024-09-17T23:32:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100800K .......... .......... .......... .......... .......... 59% 26.8M 2s 2024-09-17T23:32:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100850K .......... .......... .......... .......... .......... 59% 22.7M 2s 2024-09-17T23:32:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100900K .......... .......... .......... .......... .......... 59% 54.1M 2s 2024-09-17T23:32:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:100950K .......... .......... .......... .......... .......... 59% 22.8M 2s 2024-09-17T23:32:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101000K .......... .......... .......... .......... .......... 59% 34.9M 2s 2024-09-17T23:32:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101050K .......... .......... .......... .......... .......... 59% 24.8M 2s 2024-09-17T23:32:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101100K .......... .......... .......... .......... .......... 60% 54.9M 2s 2024-09-17T23:32:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101150K .......... .......... .......... .......... .......... 60% 24.5M 2s 2024-09-17T23:32:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101200K .......... .......... .......... .......... .......... 60% 23.0M 2s 2024-09-17T23:32:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101250K .......... .......... .......... .......... .......... 60% 37.9M 2s 2024-09-17T23:32:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101300K .......... .......... .......... .......... .......... 60% 24.1M 2s 2024-09-17T23:32:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101350K .......... .......... .......... .......... .......... 60% 54.1M 2s 2024-09-17T23:32:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101400K .......... .......... .......... .......... .......... 60% 23.9M 2s 2024-09-17T23:32:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101450K .......... .......... .......... .......... .......... 60% 34.8M 2s 2024-09-17T23:32:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101500K .......... .......... .......... .......... .......... 60% 23.4M 2s 2024-09-17T23:32:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101550K .......... .......... .......... .......... .......... 60% 25.4M 2s 2024-09-17T23:32:51.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101600K .......... .......... .......... .......... .......... 60% 24.4M 2s 2024-09-17T23:32:51.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101650K .......... .......... .......... .......... .......... 60% 53.7M 2s 2024-09-17T23:32:51.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101700K .......... .......... .......... .......... .......... 60% 35.2M 2s 2024-09-17T23:32:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101750K .......... .......... .......... .......... .......... 60% 22.9M 2s 2024-09-17T23:32:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101800K .......... .......... .......... .......... .......... 60% 22.9M 2s 2024-09-17T23:32:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101850K .......... .......... .......... .......... .......... 60% 48.3M 2s 2024-09-17T23:32:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101900K .......... .......... .......... .......... .......... 60% 32.1M 2s 2024-09-17T23:32:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:101950K .......... .......... .......... .......... .......... 60% 20.1M 2s 2024-09-17T23:32:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102000K .......... .......... .......... .......... .......... 60% 35.1M 2s 2024-09-17T23:32:51.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102050K .......... .......... .......... .......... .......... 60% 24.7M 2s 2024-09-17T23:32:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102100K .......... .......... .......... .......... .......... 60% 23.6M 2s 2024-09-17T23:32:51.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102150K .......... .......... .......... .......... .......... 60% 47.3M 2s 2024-09-17T23:32:51.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102200K .......... .......... .......... .......... .......... 60% 25.5M 2s 2024-09-17T23:32:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102250K .......... .......... .......... .......... .......... 60% 29.1M 2s 2024-09-17T23:32:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102300K .......... .......... .......... .......... .......... 60% 28.7M 2s 2024-09-17T23:32:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102350K .......... .......... .......... .......... .......... 60% 23.6M 2s 2024-09-17T23:32:51.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102400K .......... .......... .......... .......... .......... 60% 60.4M 2s 2024-09-17T23:32:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102450K .......... .......... .......... .......... .......... 60% 22.1M 2s 2024-09-17T23:32:51.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102500K .......... .......... .......... .......... .......... 60% 38.8M 2s 2024-09-17T23:32:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102550K .......... .......... .......... .......... .......... 60% 21.2M 2s 2024-09-17T23:32:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102600K .......... .......... .......... .......... .......... 60% 24.9M 2s 2024-09-17T23:32:51.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102650K .......... .......... .......... .......... .......... 60% 54.5M 2s 2024-09-17T23:32:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102700K .......... .......... .......... .......... .......... 60% 22.3M 2s 2024-09-17T23:32:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102750K .......... .......... .......... .......... .......... 60% 32.6M 2s 2024-09-17T23:32:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102800K .......... .......... .......... .......... .......... 61% 25.6M 2s 2024-09-17T23:32:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102850K .......... .......... .......... .......... .......... 61% 25.3M 2s 2024-09-17T23:32:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102900K .......... .......... .......... .......... .......... 61% 53.2M 2s 2024-09-17T23:32:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:102950K .......... .......... .......... .......... .......... 61% 23.0M 2s 2024-09-17T23:32:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103000K .......... .......... .......... .......... .......... 61% 43.2M 2s 2024-09-17T23:32:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103050K .......... .......... .......... .......... .......... 61% 22.0M 2s 2024-09-17T23:32:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103100K .......... .......... .......... .......... .......... 61% 25.8M 2s 2024-09-17T23:32:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103150K .......... .......... .......... .......... .......... 61% 23.3M 2s 2024-09-17T23:32:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103200K .......... .......... .......... .......... .......... 61% 46.0M 2s 2024-09-17T23:32:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103250K .......... .......... .......... .......... .......... 61% 25.7M 2s 2024-09-17T23:32:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103300K .......... .......... .......... .......... .......... 61% 38.4M 2s 2024-09-17T23:32:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103350K .......... .......... .......... .......... .......... 61% 24.0M 2s 2024-09-17T23:32:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103400K .......... .......... .......... .......... .......... 61% 42.3M 2s 2024-09-17T23:32:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103450K .......... .......... .......... .......... .......... 61% 25.3M 2s 2024-09-17T23:32:51.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103500K .......... .......... .......... .......... .......... 61% 37.0M 2s 2024-09-17T23:32:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103550K .......... .......... .......... .......... .......... 61% 24.7M 2s 2024-09-17T23:32:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103600K .......... .......... .......... .......... .......... 61% 23.4M 2s 2024-09-17T23:32:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103650K .......... .......... .......... .......... .......... 61% 23.6M 2s 2024-09-17T23:32:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103700K .......... .......... .......... .......... .......... 61% 49.7M 2s 2024-09-17T23:32:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103750K .......... .......... .......... .......... .......... 61% 33.8M 2s 2024-09-17T23:32:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103800K .......... .......... .......... .......... .......... 61% 27.0M 2s 2024-09-17T23:32:51.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103850K .......... .......... .......... .......... .......... 61% 23.4M 2s 2024-09-17T23:32:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103900K .......... .......... .......... .......... .......... 61% 45.6M 2s 2024-09-17T23:32:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:103950K .......... .......... .......... .......... .......... 61% 25.7M 2s 2024-09-17T23:32:51.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104000K .......... .......... .......... .......... .......... 61% 24.2M 2s 2024-09-17T23:32:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104050K .......... .......... .......... .......... .......... 61% 38.3M 2s 2024-09-17T23:32:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104100K .......... .......... .......... .......... .......... 61% 23.9M 2s 2024-09-17T23:32:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104150K .......... .......... .......... .......... .......... 61% 48.5M 2s 2024-09-17T23:32:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104200K .......... .......... .......... .......... .......... 61% 23.2M 2s 2024-09-17T23:32:51.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104250K .......... .......... .......... .......... .......... 61% 40.5M 2s 2024-09-17T23:32:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104300K .......... .......... .......... .......... .......... 61% 22.1M 2s 2024-09-17T23:32:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104350K .......... .......... .......... .......... .......... 61% 25.3M 2s 2024-09-17T23:32:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104400K .......... .......... .......... .......... .......... 61% 23.4M 2s 2024-09-17T23:32:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104450K .......... .......... .......... .......... .......... 61% 52.6M 2s 2024-09-17T23:32:51.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104500K .......... .......... .......... .......... .......... 62% 24.3M 2s 2024-09-17T23:32:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104550K .......... .......... .......... .......... .......... 62% 40.7M 2s 2024-09-17T23:32:51.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104600K .......... .......... .......... .......... .......... 62% 24.0M 2s 2024-09-17T23:32:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104650K .......... .......... .......... .......... .......... 62% 47.9M 2s 2024-09-17T23:32:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104700K .......... .......... .......... .......... .......... 62% 23.7M 2s 2024-09-17T23:32:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104750K .......... .......... .......... .......... .......... 62% 23.9M 2s 2024-09-17T23:32:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104800K .......... .......... .......... .......... .......... 62% 35.3M 2s 2024-09-17T23:32:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104850K .......... .......... .......... .......... .......... 62% 23.7M 2s 2024-09-17T23:32:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104900K .......... .......... .......... .......... .......... 62% 25.0M 2s 2024-09-17T23:32:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:104950K .......... .......... .......... .......... .......... 62% 49.6M 2s 2024-09-17T23:32:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105000K .......... .......... .......... .......... .......... 62% 24.2M 2s 2024-09-17T23:32:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105050K .......... .......... .......... .......... .......... 62% 37.2M 2s 2024-09-17T23:32:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105100K .......... .......... .......... .......... .......... 62% 24.0M 2s 2024-09-17T23:32:51.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105150K .......... .......... .......... .......... .......... 62% 24.7M 2s 2024-09-17T23:32:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105200K .......... .......... .......... .......... .......... 62% 41.5M 2s 2024-09-17T23:32:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105250K .......... .......... .......... .......... .......... 62% 26.7M 2s 2024-09-17T23:32:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105300K .......... .......... .......... .......... .......... 62% 35.1M 2s 2024-09-17T23:32:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105350K .......... .......... .......... .......... .......... 62% 25.3M 2s 2024-09-17T23:32:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105400K .......... .......... .......... .......... .......... 62% 24.1M 2s 2024-09-17T23:32:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105450K .......... .......... .......... .......... .......... 62% 50.1M 2s 2024-09-17T23:32:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105500K .......... .......... .......... .......... .......... 62% 25.9M 2s 2024-09-17T23:32:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105550K .......... .......... .......... .......... .......... 62% 21.7M 2s 2024-09-17T23:32:51.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105600K .......... .......... .......... .......... .......... 62% 37.5M 2s 2024-09-17T23:32:51.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105650K .......... .......... .......... .......... .......... 62% 24.2M 2s 2024-09-17T23:32:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105700K .......... .......... .......... .......... .......... 62% 52.7M 2s 2024-09-17T23:32:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105750K .......... .......... .......... .......... .......... 62% 27.5M 2s 2024-09-17T23:32:51.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105800K .......... .......... .......... .......... .......... 62% 28.9M 2s 2024-09-17T23:32:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105850K .......... .......... .......... .......... .......... 62% 26.5M 2s 2024-09-17T23:32:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105900K .......... .......... .......... .......... .......... 62% 42.9M 2s 2024-09-17T23:32:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:105950K .......... .......... .......... .......... .......... 62% 15.8M 2s 2024-09-17T23:32:51.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106000K .......... .......... .......... .......... .......... 62% 53.1M 2s 2024-09-17T23:32:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106050K .......... .......... .......... .......... .......... 62% 24.0M 2s 2024-09-17T23:32:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106100K .......... .......... .......... .......... .......... 62% 34.5M 2s 2024-09-17T23:32:51.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106150K .......... .......... .......... .......... .......... 63% 25.6M 2s 2024-09-17T23:32:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106200K .......... .......... .......... .......... .......... 63% 50.0M 2s 2024-09-17T23:32:51.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106250K .......... .......... .......... .......... .......... 63% 23.5M 2s 2024-09-17T23:32:51.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106300K .......... .......... .......... .......... .......... 63% 42.7M 2s 2024-09-17T23:32:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106350K .......... .......... .......... .......... .......... 63% 23.4M 2s 2024-09-17T23:32:51.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106400K .......... .......... .......... .......... .......... 63% 24.3M 2s 2024-09-17T23:32:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106450K .......... .......... .......... .......... .......... 63% 42.2M 2s 2024-09-17T23:32:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106500K .......... .......... .......... .......... .......... 63% 25.9M 2s 2024-09-17T23:32:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106550K .......... .......... .......... .......... .......... 63% 37.7M 2s 2024-09-17T23:32:51.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106600K .......... .......... .......... .......... .......... 63% 22.9M 2s 2024-09-17T23:32:51.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106650K .......... .......... .......... .......... .......... 63% 24.9M 2s 2024-09-17T23:32:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106700K .......... .......... .......... .......... .......... 63% 43.1M 2s 2024-09-17T23:32:51.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106750K .......... .......... .......... .......... .......... 63% 25.6M 2s 2024-09-17T23:32:51.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106800K .......... .......... .......... .......... .......... 63% 23.6M 2s 2024-09-17T23:32:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106850K .......... .......... .......... .......... .......... 63% 42.7M 2s 2024-09-17T23:32:51.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106900K .......... .......... .......... .......... .......... 63% 23.0M 2s 2024-09-17T23:32:51.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:106950K .......... .......... .......... .......... .......... 63% 50.6M 2s 2024-09-17T23:32:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107000K .......... .......... .......... .......... .......... 63% 23.9M 2s 2024-09-17T23:32:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107050K .......... .......... .......... .......... .......... 63% 34.9M 2s 2024-09-17T23:32:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107100K .......... .......... .......... .......... .......... 63% 24.5M 2s 2024-09-17T23:32:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107150K .......... .......... .......... .......... .......... 63% 22.8M 2s 2024-09-17T23:32:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107200K .......... .......... .......... .......... .......... 63% 23.8M 2s 2024-09-17T23:32:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107250K .......... .......... .......... .......... .......... 63% 45.0M 2s 2024-09-17T23:32:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107300K .......... .......... .......... .......... .......... 63% 26.1M 2s 2024-09-17T23:32:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107350K .......... .......... .......... .......... .......... 63% 37.9M 2s 2024-09-17T23:32:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107400K .......... .......... .......... .......... .......... 63% 23.8M 2s 2024-09-17T23:32:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107450K .......... .......... .......... .......... .......... 63% 54.8M 2s 2024-09-17T23:32:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107500K .......... .......... .......... .......... .......... 63% 22.4M 2s 2024-09-17T23:32:51.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107550K .......... .......... .......... .......... .......... 63% 25.0M 2s 2024-09-17T23:32:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107600K .......... .......... .......... .......... .......... 63% 36.4M 2s 2024-09-17T23:32:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107650K .......... .......... .......... .......... .......... 63% 23.5M 2s 2024-09-17T23:32:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107700K .......... .......... .......... .......... .......... 63% 23.5M 2s 2024-09-17T23:32:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107750K .......... .......... .......... .......... .......... 63% 56.0M 2s 2024-09-17T23:32:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107800K .......... .......... .......... .......... .......... 63% 24.0M 2s 2024-09-17T23:32:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107850K .......... .......... .......... .......... .......... 64% 39.1M 2s 2024-09-17T23:32:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107900K .......... .......... .......... .......... .......... 64% 24.2M 2s 2024-09-17T23:32:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:107950K .......... .......... .......... .......... .......... 64% 23.8M 2s 2024-09-17T23:32:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108000K .......... .......... .......... .......... .......... 64% 43.0M 2s 2024-09-17T23:32:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108050K .......... .......... .......... .......... .......... 64% 25.6M 2s 2024-09-17T23:32:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108100K .......... .......... .......... .......... .......... 64% 37.7M 2s 2024-09-17T23:32:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108150K .......... .......... .......... .......... .......... 64% 23.7M 2s 2024-09-17T23:32:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108200K .......... .......... .......... .......... .......... 64% 23.1M 2s 2024-09-17T23:32:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108250K .......... .......... .......... .......... .......... 64% 57.2M 2s 2024-09-17T23:32:51.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108300K .......... .......... .......... .......... .......... 64% 24.1M 2s 2024-09-17T23:32:51.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108350K .......... .......... .......... .......... .......... 64% 22.7M 2s 2024-09-17T23:32:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108400K .......... .......... .......... .......... .......... 64% 32.3M 2s 2024-09-17T23:32:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108450K .......... .......... .......... .......... .......... 64% 31.2M 2s 2024-09-17T23:32:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108500K .......... .......... .......... .......... .......... 64% 47.9M 2s 2024-09-17T23:32:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108550K .......... .......... .......... .......... .......... 64% 22.0M 2s 2024-09-17T23:32:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108600K .......... .......... .......... .......... .......... 64% 24.7M 2s 2024-09-17T23:32:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108650K .......... .......... .......... .......... .......... 64% 34.8M 2s 2024-09-17T23:32:51.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108700K .......... .......... .......... .......... .......... 64% 43.8M 2s 2024-09-17T23:32:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108750K .......... .......... .......... .......... .......... 64% 15.8M 2s 2024-09-17T23:32:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108800K .......... .......... .......... .......... .......... 64% 55.8M 2s 2024-09-17T23:32:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108850K .......... .......... .......... .......... .......... 64% 32.9M 2s 2024-09-17T23:32:51.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108900K .......... .......... .......... .......... .......... 64% 33.3M 2s 2024-09-17T23:32:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:108950K .......... .......... .......... .......... .......... 64% 21.4M 2s 2024-09-17T23:32:51.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109000K .......... .......... .......... .......... .......... 64% 45.6M 2s 2024-09-17T23:32:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109050K .......... .......... .......... .......... .......... 64% 32.6M 2s 2024-09-17T23:32:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109100K .......... .......... .......... .......... .......... 64% 26.4M 2s 2024-09-17T23:32:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109150K .......... .......... .......... .......... .......... 64% 17.9M 2s 2024-09-17T23:32:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109200K .......... .......... .......... .......... .......... 64% 35.6M 2s 2024-09-17T23:32:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109250K .......... .......... .......... .......... .......... 64% 48.4M 2s 2024-09-17T23:32:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109300K .......... .......... .......... .......... .......... 64% 25.5M 2s 2024-09-17T23:32:51.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109350K .......... .......... .......... .......... .......... 64% 23.0M 2s 2024-09-17T23:32:51.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109400K .......... .......... .......... .......... .......... 64% 57.5M 2s 2024-09-17T23:32:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109450K .......... .......... .......... .......... .......... 64% 18.9M 2s 2024-09-17T23:32:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109500K .......... .......... .......... .......... .......... 64% 53.7M 2s 2024-09-17T23:32:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109550K .......... .......... .......... .......... .......... 65% 24.7M 2s 2024-09-17T23:32:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109600K .......... .......... .......... .......... .......... 65% 29.0M 2s 2024-09-17T23:32:51.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109650K .......... .......... .......... .......... .......... 65% 27.8M 2s 2024-09-17T23:32:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109700K .......... .......... .......... .......... .......... 65% 34.4M 2s 2024-09-17T23:32:51.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109750K .......... .......... .......... .......... .......... 65% 31.2M 2s 2024-09-17T23:32:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109800K .......... .......... .......... .......... .......... 65% 23.8M 2s 2024-09-17T23:32:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109850K .......... .......... .......... .......... .......... 65% 38.1M 2s 2024-09-17T23:32:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109900K .......... .......... .......... .......... .......... 65% 23.7M 2s 2024-09-17T23:32:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:109950K .......... .......... .......... .......... .......... 65% 33.9M 2s 2024-09-17T23:32:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110000K .......... .......... .......... .......... .......... 65% 17.9M 2s 2024-09-17T23:32:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110050K .......... .......... .......... .......... .......... 65% 57.9M 2s 2024-09-17T23:32:51.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110100K .......... .......... .......... .......... .......... 65% 31.1M 2s 2024-09-17T23:32:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110150K .......... .......... .......... .......... .......... 65% 19.7M 2s 2024-09-17T23:32:51.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110200K .......... .......... .......... .......... .......... 65% 41.4M 2s 2024-09-17T23:32:51.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110250K .......... .......... .......... .......... .......... 65% 46.6M 2s 2024-09-17T23:32:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110300K .......... .......... .......... .......... .......... 65% 22.4M 2s 2024-09-17T23:32:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110350K .......... .......... .......... .......... .......... 65% 24.4M 2s 2024-09-17T23:32:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110400K .......... .......... .......... .......... .......... 65% 36.0M 2s 2024-09-17T23:32:51.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110450K .......... .......... .......... .......... .......... 65% 26.5M 2s 2024-09-17T23:32:51.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110500K .......... .......... .......... .......... .......... 65% 23.1M 2s 2024-09-17T23:32:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110550K .......... .......... .......... .......... .......... 65% 47.2M 2s 2024-09-17T23:32:51.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110600K .......... .......... .......... .......... .......... 65% 24.9M 2s 2024-09-17T23:32:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110650K .......... .......... .......... .......... .......... 65% 40.0M 2s 2024-09-17T23:32:51.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110700K .......... .......... .......... .......... .......... 65% 23.7M 2s 2024-09-17T23:32:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110750K .......... .......... .......... .......... .......... 65% 23.7M 2s 2024-09-17T23:32:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110800K .......... .......... .......... .......... .......... 65% 52.7M 2s 2024-09-17T23:32:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110850K .......... .......... .......... .......... .......... 65% 23.5M 2s 2024-09-17T23:32:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110900K .......... .......... .......... .......... .......... 65% 28.9M 2s 2024-09-17T23:32:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:110950K .......... .......... .......... .......... .......... 65% 29.4M 2s 2024-09-17T23:32:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111000K .......... .......... .......... .......... .......... 65% 48.1M 2s 2024-09-17T23:32:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111050K .......... .......... .......... .......... .......... 65% 22.5M 2s 2024-09-17T23:32:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111100K .......... .......... .......... .......... .......... 65% 34.1M 2s 2024-09-17T23:32:51.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111150K .......... .......... .......... .......... .......... 65% 24.1M 2s 2024-09-17T23:32:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111200K .......... .......... .......... .......... .......... 65% 24.3M 2s 2024-09-17T23:32:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111250K .......... .......... .......... .......... .......... 66% 25.2M 2s 2024-09-17T23:32:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111300K .......... .......... .......... .......... .......... 66% 47.1M 2s 2024-09-17T23:32:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111350K .......... .......... .......... .......... .......... 66% 23.9M 2s 2024-09-17T23:32:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111400K .......... .......... .......... .......... .......... 66% 44.2M 2s 2024-09-17T23:32:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111450K .......... .......... .......... .......... .......... 66% 23.2M 2s 2024-09-17T23:32:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111500K .......... .......... .......... .......... .......... 66% 46.2M 2s 2024-09-17T23:32:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111550K .......... .......... .......... .......... .......... 66% 25.7M 2s 2024-09-17T23:32:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111600K .......... .......... .......... .......... .......... 66% 23.1M 2s 2024-09-17T23:32:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111650K .......... .......... .......... .......... .......... 66% 39.4M 2s 2024-09-17T23:32:51.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111700K .......... .......... .......... .......... .......... 66% 29.6M 2s 2024-09-17T23:32:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111750K .......... .......... .......... .......... .......... 66% 18.8M 2s 2024-09-17T23:32:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111800K .......... .......... .......... .......... .......... 66% 48.3M 2s 2024-09-17T23:32:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111850K .......... .......... .......... .......... .......... 66% 25.3M 2s 2024-09-17T23:32:51.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111900K .......... .......... .......... .......... .......... 66% 36.7M 2s 2024-09-17T23:32:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:111950K .......... .......... .......... .......... .......... 66% 25.8M 2s 2024-09-17T23:32:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112000K .......... .......... .......... .......... .......... 66% 23.4M 2s 2024-09-17T23:32:51.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112050K .......... .......... .......... .......... .......... 66% 44.0M 2s 2024-09-17T23:32:51.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112100K .......... .......... .......... .......... .......... 66% 24.1M 2s 2024-09-17T23:32:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112150K .......... .......... .......... .......... .......... 66% 38.8M 2s 2024-09-17T23:32:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112200K .......... .......... .......... .......... .......... 66% 22.9M 2s 2024-09-17T23:32:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112250K .......... .......... .......... .......... .......... 66% 26.0M 2s 2024-09-17T23:32:51.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112300K .......... .......... .......... .......... .......... 66% 52.6M 2s 2024-09-17T23:32:51.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112350K .......... .......... .......... .......... .......... 66% 21.9M 2s 2024-09-17T23:32:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112400K .......... .......... .......... .......... .......... 66% 26.0M 2s 2024-09-17T23:32:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112450K .......... .......... .......... .......... .......... 66% 34.7M 2s 2024-09-17T23:32:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112500K .......... .......... .......... .......... .......... 66% 26.0M 2s 2024-09-17T23:32:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112550K .......... .......... .......... .......... .......... 66% 47.8M 2s 2024-09-17T23:32:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112600K .......... .......... .......... .......... .......... 66% 23.1M 2s 2024-09-17T23:32:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112650K .......... .......... .......... .......... .......... 66% 25.3M 2s 2024-09-17T23:32:51.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112700K .......... .......... .......... .......... .......... 66% 35.9M 2s 2024-09-17T23:32:51.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112750K .......... .......... .......... .......... .......... 66% 31.3M 2s 2024-09-17T23:32:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112800K .......... .......... .......... .......... .......... 66% 24.3M 2s 2024-09-17T23:32:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112850K .......... .......... .......... .......... .......... 66% 43.6M 2s 2024-09-17T23:32:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112900K .......... .......... .......... .......... .......... 67% 20.3M 2s 2024-09-17T23:32:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:112950K .......... .......... .......... .......... .......... 67% 26.3M 2s 2024-09-17T23:32:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113000K .......... .......... .......... .......... .......... 67% 53.7M 2s 2024-09-17T23:32:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113050K .......... .......... .......... .......... .......... 67% 37.6M 2s 2024-09-17T23:32:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113100K .......... .......... .......... .......... .......... 67% 22.3M 2s 2024-09-17T23:32:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113150K .......... .......... .......... .......... .......... 67% 25.0M 2s 2024-09-17T23:32:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113200K .......... .......... .......... .......... .......... 67% 34.4M 2s 2024-09-17T23:32:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113250K .......... .......... .......... .......... .......... 67% 25.2M 2s 2024-09-17T23:32:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113300K .......... .......... .......... .......... .......... 67% 22.2M 2s 2024-09-17T23:32:51.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113350K .......... .......... .......... .......... .......... 67% 53.8M 2s 2024-09-17T23:32:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113400K .......... .......... .......... .......... .......... 67% 23.4M 2s 2024-09-17T23:32:51.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113450K .......... .......... .......... .......... .......... 67% 40.0M 2s 2024-09-17T23:32:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113500K .......... .......... .......... .......... .......... 67% 21.9M 2s 2024-09-17T23:32:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113550K .......... .......... .......... .......... .......... 67% 24.2M 2s 2024-09-17T23:32:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113600K .......... .......... .......... .......... .......... 67% 47.5M 2s 2024-09-17T23:32:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113650K .......... .......... .......... .......... .......... 67% 24.4M 2s 2024-09-17T23:32:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113700K .......... .......... .......... .......... .......... 67% 41.4M 2s 2024-09-17T23:32:51.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113750K .......... .......... .......... .......... .......... 67% 22.0M 2s 2024-09-17T23:32:51.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113800K .......... .......... .......... .......... .......... 67% 62.7M 2s 2024-09-17T23:32:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113850K .......... .......... .......... .......... .......... 67% 22.9M 2s 2024-09-17T23:32:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113900K .......... .......... .......... .......... .......... 67% 38.9M 2s 2024-09-17T23:32:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:113950K .......... .......... .......... .......... .......... 67% 16.6M 2s 2024-09-17T23:32:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114000K .......... .......... .......... .......... .......... 67% 44.0M 2s 2024-09-17T23:32:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114050K .......... .......... .......... .......... .......... 67% 23.5M 2s 2024-09-17T23:32:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114100K .......... .......... .......... .......... .......... 67% 48.6M 2s 2024-09-17T23:32:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114150K .......... .......... .......... .......... .......... 67% 22.8M 2s 2024-09-17T23:32:51.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114200K .......... .......... .......... .......... .......... 67% 46.2M 2s 2024-09-17T23:32:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114250K .......... .......... .......... .......... .......... 67% 22.4M 2s 2024-09-17T23:32:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114300K .......... .......... .......... .......... .......... 67% 59.1M 2s 2024-09-17T23:32:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114350K .......... .......... .......... .......... .......... 67% 22.3M 2s 2024-09-17T23:32:51.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114400K .......... .......... .......... .......... .......... 67% 26.2M 2s 2024-09-17T23:32:51.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114450K .......... .......... .......... .......... .......... 67% 34.7M 2s 2024-09-17T23:32:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114500K .......... .......... .......... .......... .......... 67% 24.2M 2s 2024-09-17T23:32:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114550K .......... .......... .......... .......... .......... 67% 24.6M 2s 2024-09-17T23:32:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114600K .......... .......... .......... .......... .......... 68% 45.8M 2s 2024-09-17T23:32:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114650K .......... .......... .......... .......... .......... 68% 23.1M 2s 2024-09-17T23:32:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114700K .......... .......... .......... .......... .......... 68% 40.4M 2s 2024-09-17T23:32:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114750K .......... .......... .......... .......... .......... 68% 23.7M 2s 2024-09-17T23:32:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114800K .......... .......... .......... .......... .......... 68% 23.9M 2s 2024-09-17T23:32:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114850K .......... .......... .......... .......... .......... 68% 44.4M 2s 2024-09-17T23:32:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114900K .......... .......... .......... .......... .......... 68% 25.6M 2s 2024-09-17T23:32:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:114950K .......... .......... .......... .......... .......... 68% 9.32M 2s 2024-09-17T23:32:51.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115000K .......... .......... .......... .......... .......... 68% 147M 2s 2024-09-17T23:32:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115050K .......... .......... .......... .......... .......... 68% 196M 2s 2024-09-17T23:32:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115100K .......... .......... .......... .......... .......... 68% 92.0M 2s 2024-09-17T23:32:51.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115150K .......... .......... .......... .......... .......... 68% 23.5M 2s 2024-09-17T23:32:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115200K .......... .......... .......... .......... .......... 68% 23.0M 2s 2024-09-17T23:32:51.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115250K .......... .......... .......... .......... .......... 68% 42.7M 2s 2024-09-17T23:32:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115300K .......... .......... .......... .......... .......... 68% 23.5M 2s 2024-09-17T23:32:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115350K .......... .......... .......... .......... .......... 68% 50.0M 2s 2024-09-17T23:32:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115400K .......... .......... .......... .......... .......... 68% 23.3M 2s 2024-09-17T23:32:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115450K .......... .......... .......... .......... .......... 68% 36.1M 2s 2024-09-17T23:32:51.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115500K .......... .......... .......... .......... .......... 68% 25.4M 2s 2024-09-17T23:32:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115550K .......... .......... .......... .......... .......... 68% 22.4M 2s 2024-09-17T23:32:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115600K .......... .......... .......... .......... .......... 68% 25.2M 2s 2024-09-17T23:32:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115650K .......... .......... .......... .......... .......... 68% 52.3M 2s 2024-09-17T23:32:51.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115700K .......... .......... .......... .......... .......... 68% 24.0M 2s 2024-09-17T23:32:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115750K .......... .......... .......... .......... .......... 68% 35.4M 2s 2024-09-17T23:32:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115800K .......... .......... .......... .......... .......... 68% 29.5M 2s 2024-09-17T23:32:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115850K .......... .......... .......... .......... .......... 68% 39.4M 2s 2024-09-17T23:32:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115900K .......... .......... .......... .......... .......... 68% 22.9M 2s 2024-09-17T23:32:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:115950K .......... .......... .......... .......... .......... 68% 25.1M 2s 2024-09-17T23:32:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116000K .......... .......... .......... .......... .......... 68% 39.0M 2s 2024-09-17T23:32:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116050K .......... .......... .......... .......... .......... 68% 23.5M 2s 2024-09-17T23:32:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116100K .......... .......... .......... .......... .......... 68% 23.3M 2s 2024-09-17T23:32:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116150K .......... .......... .......... .......... .......... 68% 57.3M 2s 2024-09-17T23:32:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116200K .......... .......... .......... .......... .......... 68% 34.9M 2s 2024-09-17T23:32:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116250K .......... .......... .......... .......... .......... 68% 25.0M 2s 2024-09-17T23:32:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116300K .......... .......... .......... .......... .......... 69% 22.7M 2s 2024-09-17T23:32:51.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116350K .......... .......... .......... .......... .......... 69% 24.4M 2s 2024-09-17T23:32:51.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116400K .......... .......... .......... .......... .......... 69% 48.2M 2s 2024-09-17T23:32:51.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116450K .......... .......... .......... .......... .......... 69% 24.4M 2s 2024-09-17T23:32:51.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116500K .......... .......... .......... .......... .......... 69% 38.6M 2s 2024-09-17T23:32:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116550K .......... .......... .......... .......... .......... 69% 24.4M 2s 2024-09-17T23:32:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116600K .......... .......... .......... .......... .......... 69% 45.3M 2s 2024-09-17T23:32:51.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116650K .......... .......... .......... .......... .......... 69% 25.4M 2s 2024-09-17T23:32:51.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116700K .......... .......... .......... .......... .......... 69% 37.4M 2s 2024-09-17T23:32:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116750K .......... .......... .......... .......... .......... 69% 21.3M 2s 2024-09-17T23:32:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116800K .......... .......... .......... .......... .......... 69% 28.5M 2s 2024-09-17T23:32:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116850K .......... .......... .......... .......... .......... 69% 23.2M 2s 2024-09-17T23:32:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116900K .......... .......... .......... .......... .......... 69% 44.8M 2s 2024-09-17T23:32:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:116950K .......... .......... .......... .......... .......... 69% 25.7M 2s 2024-09-17T23:32:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117000K .......... .......... .......... .......... .......... 69% 42.2M 2s 2024-09-17T23:32:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117050K .......... .......... .......... .......... .......... 69% 22.8M 2s 2024-09-17T23:32:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117100K .......... .......... .......... .......... .......... 69% 51.9M 2s 2024-09-17T23:32:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117150K .......... .......... .......... .......... .......... 69% 23.7M 2s 2024-09-17T23:32:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117200K .......... .......... .......... .......... .......... 69% 23.2M 2s 2024-09-17T23:32:51.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117250K .......... .......... .......... .......... .......... 69% 40.0M 2s 2024-09-17T23:32:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117300K .......... .......... .......... .......... .......... 69% 22.2M 2s 2024-09-17T23:32:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117350K .......... .......... .......... .......... .......... 69% 24.1M 2s 2024-09-17T23:32:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117400K .......... .......... .......... .......... .......... 69% 55.9M 2s 2024-09-17T23:32:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117450K .......... .......... .......... .......... .......... 69% 25.7M 2s 2024-09-17T23:32:51.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117500K .......... .......... .......... .......... .......... 69% 35.6M 2s 2024-09-17T23:32:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117550K .......... .......... .......... .......... .......... 69% 22.6M 2s 2024-09-17T23:32:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117600K .......... .......... .......... .......... .......... 69% 25.0M 2s 2024-09-17T23:32:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117650K .......... .......... .......... .......... .......... 69% 52.1M 2s 2024-09-17T23:32:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117700K .......... .......... .......... .......... .......... 69% 24.6M 2s 2024-09-17T23:32:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117750K .......... .......... .......... .......... .......... 69% 39.4M 2s 2024-09-17T23:32:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117800K .......... .......... .......... .......... .......... 69% 23.5M 2s 2024-09-17T23:32:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117850K .......... .......... .......... .......... .......... 69% 23.4M 2s 2024-09-17T23:32:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117900K .......... .......... .......... .......... .......... 69% 50.3M 2s 2024-09-17T23:32:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:117950K .......... .......... .......... .......... .......... 70% 21.4M 2s 2024-09-17T23:32:51.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118000K .......... .......... .......... .......... .......... 70% 26.4M 2s 2024-09-17T23:32:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118050K .......... .......... .......... .......... .......... 70% 30.1M 2s 2024-09-17T23:32:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118100K .......... .......... .......... .......... .......... 70% 28.6M 2s 2024-09-17T23:32:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118150K .......... .......... .......... .......... .......... 70% 47.6M 2s 2024-09-17T23:32:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118200K .......... .......... .......... .......... .......... 70% 24.9M 2s 2024-09-17T23:32:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118250K .......... .......... .......... .......... .......... 70% 41.1M 2s 2024-09-17T23:32:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118300K .......... .......... .......... .......... .......... 70% 23.0M 2s 2024-09-17T23:32:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118350K .......... .......... .......... .......... .......... 70% 23.8M 2s 2024-09-17T23:32:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118400K .......... .......... .......... .......... .......... 70% 22.2M 2s 2024-09-17T23:32:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118450K .......... .......... .......... .......... .......... 70% 85.1M 2s 2024-09-17T23:32:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118500K .......... .......... .......... .......... .......... 70% 29.3M 2s 2024-09-17T23:32:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118550K .......... .......... .......... .......... .......... 70% 24.1M 2s 2024-09-17T23:32:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118600K .......... .......... .......... .......... .......... 70% 24.6M 2s 2024-09-17T23:32:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118650K .......... .......... .......... .......... .......... 70% 44.9M 2s 2024-09-17T23:32:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118700K .......... .......... .......... .......... .......... 70% 24.1M 2s 2024-09-17T23:32:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118750K .......... .......... .......... .......... .......... 70% 25.2M 2s 2024-09-17T23:32:51.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118800K .......... .......... .......... .......... .......... 70% 35.8M 2s 2024-09-17T23:32:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118850K .......... .......... .......... .......... .......... 70% 24.2M 2s 2024-09-17T23:32:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118900K .......... .......... .......... .......... .......... 70% 54.5M 2s 2024-09-17T23:32:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:118950K .......... .......... .......... .......... .......... 70% 23.7M 2s 2024-09-17T23:32:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119000K .......... .......... .......... .......... .......... 70% 32.3M 2s 2024-09-17T23:32:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119050K .......... .......... .......... .......... .......... 70% 24.7M 2s 2024-09-17T23:32:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119100K .......... .......... .......... .......... .......... 70% 25.1M 2s 2024-09-17T23:32:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119150K .......... .......... .......... .......... .......... 70% 24.4M 2s 2024-09-17T23:32:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119200K .......... .......... .......... .......... .......... 70% 60.1M 2s 2024-09-17T23:32:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119250K .......... .......... .......... .......... .......... 70% 22.6M 2s 2024-09-17T23:32:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119300K .......... .......... .......... .......... .......... 70% 38.2M 2s 2024-09-17T23:32:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119350K .......... .......... .......... .......... .......... 70% 23.1M 2s 2024-09-17T23:32:51.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119400K .......... .......... .......... .......... .......... 70% 49.7M 2s 2024-09-17T23:32:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119450K .......... .......... .......... .......... .......... 70% 25.6M 2s 2024-09-17T23:32:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119500K .......... .......... .......... .......... .......... 70% 34.2M 2s 2024-09-17T23:32:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119550K .......... .......... .......... .......... .......... 70% 22.9M 2s 2024-09-17T23:32:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119600K .......... .......... .......... .......... .......... 70% 24.4M 2s 2024-09-17T23:32:51.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119650K .......... .......... .......... .......... .......... 71% 25.7M 2s 2024-09-17T23:32:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119700K .......... .......... .......... .......... .......... 71% 43.3M 2s 2024-09-17T23:32:51.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119750K .......... .......... .......... .......... .......... 71% 26.6M 2s 2024-09-17T23:32:51.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119800K .......... .......... .......... .......... .......... 71% 34.9M 2s 2024-09-17T23:32:51.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119850K .......... .......... .......... .......... .......... 71% 24.5M 2s 2024-09-17T23:32:51.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119900K .......... .......... .......... .......... .......... 71% 57.9M 2s 2024-09-17T23:32:51.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:119950K .......... .......... .......... .......... .......... 71% 24.0M 2s 2024-09-17T23:32:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120000K .......... .......... .......... .......... .......... 71% 22.1M 2s 2024-09-17T23:32:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120050K .......... .......... .......... .......... .......... 71% 39.4M 2s 2024-09-17T23:32:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120100K .......... .......... .......... .......... .......... 71% 24.4M 2s 2024-09-17T23:32:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120150K .......... .......... .......... .......... .......... 71% 22.9M 2s 2024-09-17T23:32:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120200K .......... .......... .......... .......... .......... 71% 57.8M 2s 2024-09-17T23:32:51.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120250K .......... .......... .......... .......... .......... 71% 23.0M 2s 2024-09-17T23:32:51.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120300K .......... .......... .......... .......... .......... 71% 38.2M 2s 2024-09-17T23:32:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120350K .......... .......... .......... .......... .......... 71% 23.2M 2s 2024-09-17T23:32:51.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120400K .......... .......... .......... .......... .......... 71% 25.7M 2s 2024-09-17T23:32:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120450K .......... .......... .......... .......... .......... 71% 50.6M 2s 2024-09-17T23:32:51.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120500K .......... .......... .......... .......... .......... 71% 23.4M 2s 2024-09-17T23:32:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120550K .......... .......... .......... .......... .......... 71% 37.3M 2s 2024-09-17T23:32:51.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120600K .......... .......... .......... .......... .......... 71% 23.8M 2s 2024-09-17T23:32:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120650K .......... .......... .......... .......... .......... 71% 23.3M 2s 2024-09-17T23:32:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120700K .......... .......... .......... .......... .......... 71% 58.9M 2s 2024-09-17T23:32:51.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120750K .......... .......... .......... .......... .......... 71% 24.8M 2s 2024-09-17T23:32:51.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120800K .......... .......... .......... .......... .......... 71% 23.3M 2s 2024-09-17T23:32:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120850K .......... .......... .......... .......... .......... 71% 38.0M 2s 2024-09-17T23:32:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120900K .......... .......... .......... .......... .......... 71% 24.1M 2s 2024-09-17T23:32:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:120950K .......... .......... .......... .......... .......... 71% 46.3M 2s 2024-09-17T23:32:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121000K .......... .......... .......... .......... .......... 71% 24.4M 2s 2024-09-17T23:32:51.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121050K .......... .......... .......... .......... .......... 71% 37.3M 2s 2024-09-17T23:32:51.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121100K .......... .......... .......... .......... .......... 71% 25.5M 2s 2024-09-17T23:32:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121150K .......... .......... .......... .......... .......... 71% 23.6M 2s 2024-09-17T23:32:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121200K .......... .......... .......... .......... .......... 71% 22.8M 2s 2024-09-17T23:32:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121250K .......... .......... .......... .......... .......... 71% 52.2M 2s 2024-09-17T23:32:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121300K .......... .......... .......... .......... .......... 71% 23.9M 2s 2024-09-17T23:32:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121350K .......... .......... .......... .......... .......... 72% 40.1M 2s 2024-09-17T23:32:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121400K .......... .......... .......... .......... .......... 72% 23.9M 2s 2024-09-17T23:32:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121450K .......... .......... .......... .......... .......... 72% 44.6M 2s 2024-09-17T23:32:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121500K .......... .......... .......... .......... .......... 72% 24.1M 2s 2024-09-17T23:32:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121550K .......... .......... .......... .......... .......... 72% 23.5M 2s 2024-09-17T23:32:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121600K .......... .......... .......... .......... .......... 72% 38.8M 2s 2024-09-17T23:32:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121650K .......... .......... .......... .......... .......... 72% 24.4M 2s 2024-09-17T23:32:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121700K .......... .......... .......... .......... .......... 72% 50.4M 2s 2024-09-17T23:32:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121750K .......... .......... .......... .......... .......... 72% 24.0M 2s 2024-09-17T23:32:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121800K .......... .......... .......... .......... .......... 72% 39.8M 2s 2024-09-17T23:32:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121850K .......... .......... .......... .......... .......... 72% 23.4M 2s 2024-09-17T23:32:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121900K .......... .......... .......... .......... .......... 72% 23.5M 2s 2024-09-17T23:32:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:121950K .......... .......... .......... .......... .......... 72% 22.5M 2s 2024-09-17T23:32:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122000K .......... .......... .......... .......... .......... 72% 51.6M 2s 2024-09-17T23:32:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122050K .......... .......... .......... .......... .......... 72% 23.2M 2s 2024-09-17T23:32:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122100K .......... .......... .......... .......... .......... 72% 40.2M 2s 2024-09-17T23:32:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122150K .......... .......... .......... .......... .......... 72% 24.1M 2s 2024-09-17T23:32:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122200K .......... .......... .......... .......... .......... 72% 61.7M 2s 2024-09-17T23:32:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122250K .......... .......... .......... .......... .......... 72% 22.4M 2s 2024-09-17T23:32:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122300K .......... .......... .......... .......... .......... 72% 36.3M 2s 2024-09-17T23:32:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122350K .......... .......... .......... .......... .......... 72% 22.7M 2s 2024-09-17T23:32:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122400K .......... .......... .......... .......... .......... 72% 24.4M 2s 2024-09-17T23:32:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122450K .......... .......... .......... .......... .......... 72% 25.2M 2s 2024-09-17T23:32:51.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122500K .......... .......... .......... .......... .......... 72% 51.2M 2s 2024-09-17T23:32:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122550K .......... .......... .......... .......... .......... 72% 22.8M 2s 2024-09-17T23:32:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122600K .......... .......... .......... .......... .......... 72% 43.3M 2s 2024-09-17T23:32:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122650K .......... .......... .......... .......... .......... 72% 22.4M 2s 2024-09-17T23:32:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122700K .......... .......... .......... .......... .......... 72% 51.6M 2s 2024-09-17T23:32:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122750K .......... .......... .......... .......... .......... 72% 24.6M 2s 2024-09-17T23:32:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122800K .......... .......... .......... .......... .......... 72% 24.5M 2s 2024-09-17T23:32:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122850K .......... .......... .......... .......... .......... 72% 34.9M 2s 2024-09-17T23:32:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122900K .......... .......... .......... .......... .......... 72% 24.4M 2s 2024-09-17T23:32:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:122950K .......... .......... .......... .......... .......... 72% 23.7M 2s 2024-09-17T23:32:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123000K .......... .......... .......... .......... .......... 72% 56.2M 2s 2024-09-17T23:32:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123050K .......... .......... .......... .......... .......... 73% 25.5M 2s 2024-09-17T23:32:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123100K .......... .......... .......... .......... .......... 73% 33.4M 2s 2024-09-17T23:32:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123150K .......... .......... .......... .......... .......... 73% 23.4M 2s 2024-09-17T23:32:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123200K .......... .......... .......... .......... .......... 73% 24.4M 2s 2024-09-17T23:32:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123250K .......... .......... .......... .......... .......... 73% 49.7M 2s 2024-09-17T23:32:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123300K .......... .......... .......... .......... .......... 73% 24.6M 2s 2024-09-17T23:32:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123350K .......... .......... .......... .......... .......... 73% 38.4M 2s 2024-09-17T23:32:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123400K .......... .......... .......... .......... .......... 73% 23.0M 2s 2024-09-17T23:32:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123450K .......... .......... .......... .......... .......... 73% 58.5M 2s 2024-09-17T23:32:51.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123500K .......... .......... .......... .......... .......... 73% 22.7M 2s 2024-09-17T23:32:51.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123550K .......... .......... .......... .......... .......... 73% 24.7M 2s 2024-09-17T23:32:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123600K .......... .......... .......... .......... .......... 73% 23.3M 2s 2024-09-17T23:32:51.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123650K .......... .......... .......... .......... .......... 73% 37.4M 2s 2024-09-17T23:32:51.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123700K .......... .......... .......... .......... .......... 73% 24.0M 2s 2024-09-17T23:32:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123750K .......... .......... .......... .......... .......... 73% 48.1M 2s 2024-09-17T23:32:51.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123800K .......... .......... .......... .......... .......... 73% 25.3M 2s 2024-09-17T23:32:51.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123850K .......... .......... .......... .......... .......... 73% 38.5M 2s 2024-09-17T23:32:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123900K .......... .......... .......... .......... .......... 73% 24.4M 2s 2024-09-17T23:32:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:123950K .......... .......... .......... .......... .......... 73% 23.6M 2s 2024-09-17T23:32:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124000K .......... .......... .......... .......... .......... 73% 46.5M 2s 2024-09-17T23:32:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124050K .......... .......... .......... .......... .......... 73% 24.0M 2s 2024-09-17T23:32:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124100K .......... .......... .......... .......... .......... 73% 35.5M 2s 2024-09-17T23:32:51.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124150K .......... .......... .......... .......... .......... 73% 23.4M 2s 2024-09-17T23:32:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124200K .......... .......... .......... .......... .......... 73% 24.5M 2s 2024-09-17T23:32:51.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124250K .......... .......... .......... .......... .......... 73% 63.5M 2s 2024-09-17T23:32:51.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124300K .......... .......... .......... .......... .......... 73% 22.9M 2s 2024-09-17T23:32:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124350K .......... .......... .......... .......... .......... 73% 23.5M 2s 2024-09-17T23:32:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124400K .......... .......... .......... .......... .......... 73% 43.2M 2s 2024-09-17T23:32:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124450K .......... .......... .......... .......... .......... 73% 23.9M 2s 2024-09-17T23:32:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124500K .......... .......... .......... .......... .......... 73% 50.6M 2s 2024-09-17T23:32:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124550K .......... .......... .......... .......... .......... 73% 23.4M 2s 2024-09-17T23:32:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124600K .......... .......... .......... .......... .......... 73% 42.3M 2s 2024-09-17T23:32:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124650K .......... .......... .......... .......... .......... 73% 21.5M 1s 2024-09-17T23:32:51.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124700K .......... .......... .......... .......... .......... 74% 25.1M 1s 2024-09-17T23:32:51.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124750K .......... .......... .......... .......... .......... 74% 23.0M 1s 2024-09-17T23:32:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124800K .......... .......... .......... .......... .......... 74% 53.3M 1s 2024-09-17T23:32:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124850K .......... .......... .......... .......... .......... 74% 27.7M 1s 2024-09-17T23:32:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124900K .......... .......... .......... .......... .......... 74% 30.3M 1s 2024-09-17T23:32:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:124950K .......... .......... .......... .......... .......... 74% 30.1M 1s 2024-09-17T23:32:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125000K .......... .......... .......... .......... .......... 74% 36.2M 1s 2024-09-17T23:32:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125050K .......... .......... .......... .......... .......... 74% 27.9M 1s 2024-09-17T23:32:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125100K .......... .......... .......... .......... .......... 74% 29.5M 1s 2024-09-17T23:32:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125150K .......... .......... .......... .......... .......... 74% 17.7M 1s 2024-09-17T23:32:51.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125200K .......... .......... .......... .......... .......... 74% 44.9M 1s 2024-09-17T23:32:51.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125250K .......... .......... .......... .......... .......... 74% 35.1M 1s 2024-09-17T23:32:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125300K .......... .......... .......... .......... .......... 74% 29.4M 1s 2024-09-17T23:32:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125350K .......... .......... .......... .......... .......... 74% 26.5M 1s 2024-09-17T23:32:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125400K .......... .......... .......... .......... .......... 74% 30.6M 1s 2024-09-17T23:32:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125450K .......... .......... .......... .......... .......... 74% 23.7M 1s 2024-09-17T23:32:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125500K .......... .......... .......... .......... .......... 74% 70.4M 1s 2024-09-17T23:32:51.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125550K .......... .......... .......... .......... .......... 74% 21.4M 1s 2024-09-17T23:32:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125600K .......... .......... .......... .......... .......... 74% 23.7M 1s 2024-09-17T23:32:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125650K .......... .......... .......... .......... .......... 74% 41.0M 1s 2024-09-17T23:32:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125700K .......... .......... .......... .......... .......... 74% 26.4M 1s 2024-09-17T23:32:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125750K .......... .......... .......... .......... .......... 74% 20.6M 1s 2024-09-17T23:32:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125800K .......... .......... .......... .......... .......... 74% 56.9M 1s 2024-09-17T23:32:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125850K .......... .......... .......... .......... .......... 74% 36.6M 1s 2024-09-17T23:32:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125900K .......... .......... .......... .......... .......... 74% 34.2M 1s 2024-09-17T23:32:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:125950K .......... .......... .......... .......... .......... 74% 17.7M 1s 2024-09-17T23:32:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126000K .......... .......... .......... .......... .......... 74% 36.3M 1s 2024-09-17T23:32:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126050K .......... .......... .......... .......... .......... 74% 35.0M 1s 2024-09-17T23:32:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126100K .......... .......... .......... .......... .......... 74% 27.8M 1s 2024-09-17T23:32:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126150K .......... .......... .......... .......... .......... 74% 28.5M 1s 2024-09-17T23:32:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126200K .......... .......... .......... .......... .......... 74% 24.0M 1s 2024-09-17T23:32:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126250K .......... .......... .......... .......... .......... 74% 50.7M 1s 2024-09-17T23:32:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126300K .......... .......... .......... .......... .......... 74% 23.8M 1s 2024-09-17T23:32:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126350K .......... .......... .......... .......... .......... 74% 22.2M 1s 2024-09-17T23:32:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126400K .......... .......... .......... .......... .......... 75% 44.2M 1s 2024-09-17T23:32:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126450K .......... .......... .......... .......... .......... 75% 23.7M 1s 2024-09-17T23:32:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126500K .......... .......... .......... .......... .......... 75% 23.1M 1s 2024-09-17T23:32:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126550K .......... .......... .......... .......... .......... 75% 48.3M 1s 2024-09-17T23:32:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126600K .......... .......... .......... .......... .......... 75% 25.6M 1s 2024-09-17T23:32:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126650K .......... .......... .......... .......... .......... 75% 23.3M 1s 2024-09-17T23:32:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126700K .......... .......... .......... .......... .......... 75% 29.9M 1s 2024-09-17T23:32:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126750K .......... .......... .......... .......... .......... 75% 40.9M 1s 2024-09-17T23:32:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126800K .......... .......... .......... .......... .......... 75% 32.2M 1s 2024-09-17T23:32:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126850K .......... .......... .......... .......... .......... 75% 29.7M 1s 2024-09-17T23:32:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126900K .......... .......... .......... .......... .......... 75% 27.5M 1s 2024-09-17T23:32:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:126950K .......... .......... .......... .......... .......... 75% 24.6M 1s 2024-09-17T23:32:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127000K .......... .......... .......... .......... .......... 75% 24.2M 1s 2024-09-17T23:32:51.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127050K .......... .......... .......... .......... .......... 75% 52.9M 1s 2024-09-17T23:32:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127100K .......... .......... .......... .......... .......... 75% 23.7M 1s 2024-09-17T23:32:51.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127150K .......... .......... .......... .......... .......... 75% 25.1M 1s 2024-09-17T23:32:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127200K .......... .......... .......... .......... .......... 75% 35.4M 1s 2024-09-17T23:32:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127250K .......... .......... .......... .......... .......... 75% 25.9M 1s 2024-09-17T23:32:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127300K .......... .......... .......... .......... .......... 75% 49.9M 1s 2024-09-17T23:32:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127350K .......... .......... .......... .......... .......... 75% 23.6M 1s 2024-09-17T23:32:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127400K .......... .......... .......... .......... .......... 75% 23.0M 1s 2024-09-17T23:32:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127450K .......... .......... .......... .......... .......... 75% 38.3M 1s 2024-09-17T23:32:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127500K .......... .......... .......... .......... .......... 75% 25.8M 1s 2024-09-17T23:32:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127550K .......... .......... .......... .......... .......... 75% 22.8M 1s 2024-09-17T23:32:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127600K .......... .......... .......... .......... .......... 75% 52.7M 1s 2024-09-17T23:32:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127650K .......... .......... .......... .......... .......... 75% 22.9M 1s 2024-09-17T23:32:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127700K .......... .......... .......... .......... .......... 75% 41.2M 1s 2024-09-17T23:32:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127750K .......... .......... .......... .......... .......... 75% 23.1M 1s 2024-09-17T23:32:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127800K .......... .......... .......... .......... .......... 75% 50.7M 1s 2024-09-17T23:32:51.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127850K .......... .......... .......... .......... .......... 75% 23.2M 1s 2024-09-17T23:32:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127900K .......... .......... .......... .......... .......... 75% 40.7M 1s 2024-09-17T23:32:51.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:127950K .......... .......... .......... .......... .......... 75% 23.7M 1s 2024-09-17T23:32:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128000K .......... .......... .......... .......... .......... 75% 24.1M 1s 2024-09-17T23:32:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128050K .......... .......... .......... .......... .......... 75% 21.7M 1s 2024-09-17T23:32:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128100K .......... .......... .......... .......... .......... 76% 54.9M 1s 2024-09-17T23:32:51.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128150K .......... .......... .......... .......... .......... 76% 23.9M 1s 2024-09-17T23:32:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128200K .......... .......... .......... .......... .......... 76% 37.7M 1s 2024-09-17T23:32:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128250K .......... .......... .......... .......... .......... 76% 23.2M 1s 2024-09-17T23:32:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128300K .......... .......... .......... .......... .......... 76% 57.1M 1s 2024-09-17T23:32:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128350K .......... .......... .......... .......... .......... 76% 23.1M 1s 2024-09-17T23:32:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128400K .......... .......... .......... .......... .......... 76% 23.2M 1s 2024-09-17T23:32:52.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128450K .......... .......... .......... .......... .......... 76% 39.4M 1s 2024-09-17T23:32:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128500K .......... .......... .......... .......... .......... 76% 34.2M 1s 2024-09-17T23:32:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128550K .......... .......... .......... .......... .......... 76% 35.8M 1s 2024-09-17T23:32:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128600K .......... .......... .......... .......... .......... 76% 21.9M 1s 2024-09-17T23:32:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128650K .......... .......... .......... .......... .......... 76% 34.6M 1s 2024-09-17T23:32:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128700K .......... .......... .......... .......... .......... 76% 15.8M 1s 2024-09-17T23:32:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128750K .......... .......... .......... .......... .......... 76% 57.6M 1s 2024-09-17T23:32:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128800K .......... .......... .......... .......... .......... 76% 36.6M 1s 2024-09-17T23:32:52.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128850K .......... .......... .......... .......... .......... 76% 30.9M 1s 2024-09-17T23:32:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128900K .......... .......... .......... .......... .......... 76% 25.4M 1s 2024-09-17T23:32:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:128950K .......... .......... .......... .......... .......... 76% 34.0M 1s 2024-09-17T23:32:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129000K .......... .......... .......... .......... .......... 76% 22.4M 1s 2024-09-17T23:32:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129050K .......... .......... .......... .......... .......... 76% 49.0M 1s 2024-09-17T23:32:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129100K .......... .......... .......... .......... .......... 76% 24.8M 1s 2024-09-17T23:32:52.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129150K .......... .......... .......... .......... .......... 76% 25.0M 1s 2024-09-17T23:32:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129200K .......... .......... .......... .......... .......... 76% 39.1M 1s 2024-09-17T23:32:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129250K .......... .......... .......... .......... .......... 76% 23.2M 1s 2024-09-17T23:32:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129300K .......... .......... .......... .......... .......... 76% 24.1M 1s 2024-09-17T23:32:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129350K .......... .......... .......... .......... .......... 76% 45.5M 1s 2024-09-17T23:32:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129400K .......... .......... .......... .......... .......... 76% 25.7M 1s 2024-09-17T23:32:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129450K .......... .......... .......... .......... .......... 76% 24.7M 1s 2024-09-17T23:32:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129500K .......... .......... .......... .......... .......... 76% 30.3M 1s 2024-09-17T23:32:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129550K .......... .......... .......... .......... .......... 76% 36.6M 1s 2024-09-17T23:32:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129600K .......... .......... .......... .......... .......... 76% 33.9M 1s 2024-09-17T23:32:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129650K .......... .......... .......... .......... .......... 76% 25.2M 1s 2024-09-17T23:32:52.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129700K .......... .......... .......... .......... .......... 76% 21.3M 1s 2024-09-17T23:32:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129750K .......... .......... .......... .......... .......... 77% 37.8M 1s 2024-09-17T23:32:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129800K .......... .......... .......... .......... .......... 77% 22.6M 1s 2024-09-17T23:32:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129850K .......... .......... .......... .......... .......... 77% 54.7M 1s 2024-09-17T23:32:52.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129900K .......... .......... .......... .......... .......... 77% 28.9M 1s 2024-09-17T23:32:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:129950K .......... .......... .......... .......... .......... 77% 22.2M 1s 2024-09-17T23:32:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130000K .......... .......... .......... .......... .......... 77% 34.1M 1s 2024-09-17T23:32:52.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130050K .......... .......... .......... .......... .......... 77% 25.9M 1s 2024-09-17T23:32:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130100K .......... .......... .......... .......... .......... 77% 41.2M 1s 2024-09-17T23:32:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130150K .......... .......... .......... .......... .......... 77% 29.0M 1s 2024-09-17T23:32:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130200K .......... .......... .......... .......... .......... 77% 34.9M 1s 2024-09-17T23:32:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130250K .......... .......... .......... .......... .......... 77% 32.1M 1s 2024-09-17T23:32:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130300K .......... .......... .......... .......... .......... 77% 17.7M 1s 2024-09-17T23:32:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130350K .......... .......... .......... .......... .......... 77% 25.4M 1s 2024-09-17T23:32:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130400K .......... .......... .......... .......... .......... 77% 48.5M 1s 2024-09-17T23:32:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130450K .......... .......... .......... .......... .......... 77% 23.5M 1s 2024-09-17T23:32:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130500K .......... .......... .......... .......... .......... 77% 49.0M 1s 2024-09-17T23:32:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130550K .......... .......... .......... .......... .......... 77% 21.3M 1s 2024-09-17T23:32:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130600K .......... .......... .......... .......... .......... 77% 52.8M 1s 2024-09-17T23:32:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130650K .......... .......... .......... .......... .......... 77% 23.6M 1s 2024-09-17T23:32:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130700K .......... .......... .......... .......... .......... 77% 40.6M 1s 2024-09-17T23:32:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130750K .......... .......... .......... .......... .......... 77% 23.5M 1s 2024-09-17T23:32:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130800K .......... .......... .......... .......... .......... 77% 23.3M 1s 2024-09-17T23:32:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130850K .......... .......... .......... .......... .......... 77% 23.3M 1s 2024-09-17T23:32:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130900K .......... .......... .......... .......... .......... 77% 53.7M 1s 2024-09-17T23:32:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:130950K .......... .......... .......... .......... .......... 77% 30.7M 1s 2024-09-17T23:32:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131000K .......... .......... .......... .......... .......... 77% 26.3M 1s 2024-09-17T23:32:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131050K .......... .......... .......... .......... .......... 77% 23.8M 1s 2024-09-17T23:32:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131100K .......... .......... .......... .......... .......... 77% 56.2M 1s 2024-09-17T23:32:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131150K .......... .......... .......... .......... .......... 77% 23.8M 1s 2024-09-17T23:32:52.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131200K .......... .......... .......... .......... .......... 77% 23.8M 1s 2024-09-17T23:32:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131250K .......... .......... .......... .......... .......... 77% 39.5M 1s 2024-09-17T23:32:52.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131300K .......... .......... .......... .......... .......... 77% 26.8M 1s 2024-09-17T23:32:52.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131350K .......... .......... .......... .......... .......... 77% 37.2M 1s 2024-09-17T23:32:52.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131400K .......... .......... .......... .......... .......... 77% 22.8M 1s 2024-09-17T23:32:52.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131450K .......... .......... .......... .......... .......... 78% 40.7M 1s 2024-09-17T23:32:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131500K .......... .......... .......... .......... .......... 78% 23.3M 1s 2024-09-17T23:32:52.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131550K .......... .......... .......... .......... .......... 78% 26.9M 1s 2024-09-17T23:32:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131600K .......... .......... .......... .......... .......... 78% 22.5M 1s 2024-09-17T23:32:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131650K .......... .......... .......... .......... .......... 78% 55.2M 1s 2024-09-17T23:32:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131700K .......... .......... .......... .......... .......... 78% 23.3M 1s 2024-09-17T23:32:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131750K .......... .......... .......... .......... .......... 78% 33.2M 1s 2024-09-17T23:32:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131800K .......... .......... .......... .......... .......... 78% 25.0M 1s 2024-09-17T23:32:52.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131850K .......... .......... .......... .......... .......... 78% 48.7M 1s 2024-09-17T23:32:52.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131900K .......... .......... .......... .......... .......... 78% 23.9M 1s 2024-09-17T23:32:52.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:131950K .......... .......... .......... .......... .......... 78% 23.9M 1s 2024-09-17T23:32:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132000K .......... .......... .......... .......... .......... 78% 36.1M 1s 2024-09-17T23:32:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132050K .......... .......... .......... .......... .......... 78% 23.0M 1s 2024-09-17T23:32:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132100K .......... .......... .......... .......... .......... 78% 24.8M 1s 2024-09-17T23:32:52.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132150K .......... .......... .......... .......... .......... 78% 54.1M 1s 2024-09-17T23:32:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132200K .......... .......... .......... .......... .......... 78% 23.4M 1s 2024-09-17T23:32:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132250K .......... .......... .......... .......... .......... 78% 40.6M 1s 2024-09-17T23:32:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132300K .......... .......... .......... .......... .......... 78% 24.2M 1s 2024-09-17T23:32:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132350K .......... .......... .......... .......... .......... 78% 22.5M 1s 2024-09-17T23:32:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132400K .......... .......... .......... .......... .......... 78% 50.4M 1s 2024-09-17T23:32:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132450K .......... .......... .......... .......... .......... 78% 25.0M 1s 2024-09-17T23:32:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132500K .......... .......... .......... .......... .......... 78% 37.8M 1s 2024-09-17T23:32:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132550K .......... .......... .......... .......... .......... 78% 24.1M 1s 2024-09-17T23:32:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132600K .......... .......... .......... .......... .......... 78% 23.3M 1s 2024-09-17T23:32:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132650K .......... .......... .......... .......... .......... 78% 48.8M 1s 2024-09-17T23:32:52.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132700K .......... .......... .......... .......... .......... 78% 24.9M 1s 2024-09-17T23:32:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132750K .......... .......... .......... .......... .......... 78% 23.5M 1s 2024-09-17T23:32:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132800K .......... .......... .......... .......... .......... 78% 38.1M 1s 2024-09-17T23:32:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132850K .......... .......... .......... .......... .......... 78% 24.3M 1s 2024-09-17T23:32:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132900K .......... .......... .......... .......... .......... 78% 51.6M 1s 2024-09-17T23:32:52.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:132950K .......... .......... .......... .......... .......... 78% 24.8M 1s 2024-09-17T23:32:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133000K .......... .......... .......... .......... .......... 78% 37.8M 1s 2024-09-17T23:32:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133050K .......... .......... .......... .......... .......... 78% 24.5M 1s 2024-09-17T23:32:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133100K .......... .......... .......... .......... .......... 78% 27.0M 1s 2024-09-17T23:32:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133150K .......... .......... .......... .......... .......... 79% 20.5M 1s 2024-09-17T23:32:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133200K .......... .......... .......... .......... .......... 79% 44.3M 1s 2024-09-17T23:32:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133250K .......... .......... .......... .......... .......... 79% 25.7M 1s 2024-09-17T23:32:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133300K .......... .......... .......... .......... .......... 79% 37.5M 1s 2024-09-17T23:32:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133350K .......... .......... .......... .......... .......... 79% 23.5M 1s 2024-09-17T23:32:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133400K .......... .......... .......... .......... .......... 79% 48.4M 1s 2024-09-17T23:32:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133450K .......... .......... .......... .......... .......... 79% 25.3M 1s 2024-09-17T23:32:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133500K .......... .......... .......... .......... .......... 79% 33.9M 1s 2024-09-17T23:32:52.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133550K .......... .......... .......... .......... .......... 79% 26.2M 1s 2024-09-17T23:32:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133600K .......... .......... .......... .......... .......... 79% 22.8M 1s 2024-09-17T23:32:52.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133650K .......... .......... .......... .......... .......... 79% 24.7M 1s 2024-09-17T23:32:52.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133700K .......... .......... .......... .......... .......... 79% 51.8M 1s 2024-09-17T23:32:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133750K .......... .......... .......... .......... .......... 79% 22.5M 1s 2024-09-17T23:32:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133800K .......... .......... .......... .......... .......... 79% 41.8M 1s 2024-09-17T23:32:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133850K .......... .......... .......... .......... .......... 79% 23.0M 1s 2024-09-17T23:32:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133900K .......... .......... .......... .......... .......... 79% 57.1M 1s 2024-09-17T23:32:52.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:133950K .......... .......... .......... .......... .......... 79% 22.7M 1s 2024-09-17T23:32:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134000K .......... .......... .......... .......... .......... 79% 23.5M 1s 2024-09-17T23:32:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134050K .......... .......... .......... .......... .......... 79% 37.5M 1s 2024-09-17T23:32:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134100K .......... .......... .......... .......... .......... 79% 25.7M 1s 2024-09-17T23:32:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134150K .......... .......... .......... .......... .......... 79% 45.7M 1s 2024-09-17T23:32:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134200K .......... .......... .......... .......... .......... 79% 25.2M 1s 2024-09-17T23:32:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134250K .......... .......... .......... .......... .......... 79% 37.5M 1s 2024-09-17T23:32:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134300K .......... .......... .......... .......... .......... 79% 22.5M 1s 2024-09-17T23:32:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134350K .......... .......... .......... .......... .......... 79% 23.9M 1s 2024-09-17T23:32:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134400K .......... .......... .......... .......... .......... 79% 26.3M 1s 2024-09-17T23:32:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134450K .......... .......... .......... .......... .......... 79% 49.8M 1s 2024-09-17T23:32:52.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134500K .......... .......... .......... .......... .......... 79% 23.0M 1s 2024-09-17T23:32:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134550K .......... .......... .......... .......... .......... 79% 40.5M 1s 2024-09-17T23:32:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134600K .......... .......... .......... .......... .......... 79% 25.7M 1s 2024-09-17T23:32:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134650K .......... .......... .......... .......... .......... 79% 40.5M 1s 2024-09-17T23:32:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134700K .......... .......... .......... .......... .......... 79% 24.6M 1s 2024-09-17T23:32:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134750K .......... .......... .......... .......... .......... 79% 24.3M 1s 2024-09-17T23:32:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134800K .......... .......... .......... .......... .......... 79% 36.2M 1s 2024-09-17T23:32:52.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134850K .......... .......... .......... .......... .......... 80% 22.7M 1s 2024-09-17T23:32:52.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134900K .......... .......... .......... .......... .......... 80% 24.9M 1s 2024-09-17T23:32:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:134950K .......... .......... .......... .......... .......... 80% 56.5M 1s 2024-09-17T23:32:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135000K .......... .......... .......... .......... .......... 80% 23.1M 1s 2024-09-17T23:32:52.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135050K .......... .......... .......... .......... .......... 80% 40.1M 1s 2024-09-17T23:32:52.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135100K .......... .......... .......... .......... .......... 80% 24.6M 1s 2024-09-17T23:32:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135150K .......... .......... .......... .......... .......... 80% 23.1M 1s 2024-09-17T23:32:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135200K .......... .......... .......... .......... .......... 80% 46.5M 1s 2024-09-17T23:32:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135250K .......... .......... .......... .......... .......... 80% 25.7M 1s 2024-09-17T23:32:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135300K .......... .......... .......... .......... .......... 80% 36.2M 1s 2024-09-17T23:32:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135350K .......... .......... .......... .......... .......... 80% 23.3M 1s 2024-09-17T23:32:52.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135400K .......... .......... .......... .......... .......... 80% 24.6M 1s 2024-09-17T23:32:52.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135450K .......... .......... .......... .......... .......... 80% 52.1M 1s 2024-09-17T23:32:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135500K .......... .......... .......... .......... .......... 80% 38.3M 1s 2024-09-17T23:32:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135550K .......... .......... .......... .......... .......... 80% 17.3M 1s 2024-09-17T23:32:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135600K .......... .......... .......... .......... .......... 80% 40.9M 1s 2024-09-17T23:32:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135650K .......... .......... .......... .......... .......... 80% 23.5M 1s 2024-09-17T23:32:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135700K .......... .......... .......... .......... .......... 80% 57.5M 1s 2024-09-17T23:32:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135750K .......... .......... .......... .......... .......... 80% 22.3M 1s 2024-09-17T23:32:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135800K .......... .......... .......... .......... .......... 80% 40.7M 1s 2024-09-17T23:32:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135850K .......... .......... .......... .......... .......... 80% 23.8M 1s 2024-09-17T23:32:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135900K .......... .......... .......... .......... .......... 80% 51.1M 1s 2024-09-17T23:32:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:135950K .......... .......... .......... .......... .......... 80% 14.8M 1s 2024-09-17T23:32:52.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136000K .......... .......... .......... .......... .......... 80% 58.8M 1s 2024-09-17T23:32:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136050K .......... .......... .......... .......... .......... 80% 32.3M 1s 2024-09-17T23:32:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136100K .......... .......... .......... .......... .......... 80% 25.9M 1s 2024-09-17T23:32:52.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136150K .......... .......... .......... .......... .......... 80% 24.6M 1s 2024-09-17T23:32:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136200K .......... .......... .......... .......... .......... 80% 52.4M 1s 2024-09-17T23:32:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136250K .......... .......... .......... .......... .......... 80% 23.6M 1s 2024-09-17T23:32:52.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136300K .......... .......... .......... .......... .......... 80% 38.0M 1s 2024-09-17T23:32:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136350K .......... .......... .......... .......... .......... 80% 24.4M 1s 2024-09-17T23:32:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136400K .......... .......... .......... .......... .......... 80% 23.3M 1s 2024-09-17T23:32:52.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136450K .......... .......... .......... .......... .......... 80% 54.6M 1s 2024-09-17T23:32:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136500K .......... .......... .......... .......... .......... 81% 23.8M 1s 2024-09-17T23:32:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136550K .......... .......... .......... .......... .......... 81% 35.2M 1s 2024-09-17T23:32:52.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136600K .......... .......... .......... .......... .......... 81% 23.1M 1s 2024-09-17T23:32:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136650K .......... .......... .......... .......... .......... 81% 24.9M 1s 2024-09-17T23:32:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136700K .......... .......... .......... .......... .......... 81% 56.1M 1s 2024-09-17T23:32:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136750K .......... .......... .......... .......... .......... 81% 22.7M 1s 2024-09-17T23:32:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136800K .......... .......... .......... .......... .......... 81% 24.2M 1s 2024-09-17T23:32:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136850K .......... .......... .......... .......... .......... 81% 41.2M 1s 2024-09-17T23:32:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136900K .......... .......... .......... .......... .......... 81% 23.7M 1s 2024-09-17T23:32:52.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:136950K .......... .......... .......... .......... .......... 81% 54.3M 1s 2024-09-17T23:32:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137000K .......... .......... .......... .......... .......... 81% 22.9M 1s 2024-09-17T23:32:52.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137050K .......... .......... .......... .......... .......... 81% 40.9M 1s 2024-09-17T23:32:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137100K .......... .......... .......... .......... .......... 81% 23.2M 1s 2024-09-17T23:32:52.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137150K .......... .......... .......... .......... .......... 81% 21.7M 1s 2024-09-17T23:32:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137200K .......... .......... .......... .......... .......... 81% 24.5M 1s 2024-09-17T23:32:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137250K .......... .......... .......... .......... .......... 81% 47.4M 1s 2024-09-17T23:32:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137300K .......... .......... .......... .......... .......... 81% 26.2M 1s 2024-09-17T23:32:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137350K .......... .......... .......... .......... .......... 81% 36.5M 1s 2024-09-17T23:32:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137400K .......... .......... .......... .......... .......... 81% 23.8M 1s 2024-09-17T23:32:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137450K .......... .......... .......... .......... .......... 81% 59.1M 1s 2024-09-17T23:32:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137500K .......... .......... .......... .......... .......... 81% 23.2M 1s 2024-09-17T23:32:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137550K .......... .......... .......... .......... .......... 81% 23.8M 1s 2024-09-17T23:32:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137600K .......... .......... .......... .......... .......... 81% 35.3M 1s 2024-09-17T23:32:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137650K .......... .......... .......... .......... .......... 81% 25.7M 1s 2024-09-17T23:32:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137700K .......... .......... .......... .......... .......... 81% 23.5M 1s 2024-09-17T23:32:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137750K .......... .......... .......... .......... .......... 81% 45.8M 1s 2024-09-17T23:32:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137800K .......... .......... .......... .......... .......... 81% 22.1M 1s 2024-09-17T23:32:52.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137850K .......... .......... .......... .......... .......... 81% 42.0M 1s 2024-09-17T23:32:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137900K .......... .......... .......... .......... .......... 81% 26.5M 1s 2024-09-17T23:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:137950K .......... .......... .......... .......... .......... 81% 22.7M 1s 2024-09-17T23:32:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138000K .......... .......... .......... .......... .......... 81% 50.2M 1s 2024-09-17T23:32:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138050K .......... .......... .......... .......... .......... 81% 35.9M 1s 2024-09-17T23:32:52.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138100K .......... .......... .......... .......... .......... 81% 26.1M 1s 2024-09-17T23:32:52.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138150K .......... .......... .......... .......... .......... 81% 23.6M 1s 2024-09-17T23:32:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138200K .......... .......... .......... .......... .......... 82% 22.1M 1s 2024-09-17T23:32:52.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138250K .......... .......... .......... .......... .......... 82% 58.2M 1s 2024-09-17T23:32:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138300K .......... .......... .......... .......... .......... 82% 23.3M 1s 2024-09-17T23:32:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138350K .......... .......... .......... .......... .......... 82% 25.0M 1s 2024-09-17T23:32:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138400K .......... .......... .......... .......... .......... 82% 38.9M 1s 2024-09-17T23:32:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138450K .......... .......... .......... .......... .......... 82% 24.2M 1s 2024-09-17T23:32:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138500K .......... .......... .......... .......... .......... 82% 48.8M 1s 2024-09-17T23:32:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138550K .......... .......... .......... .......... .......... 82% 29.3M 1s 2024-09-17T23:32:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138600K .......... .......... .......... .......... .......... 82% 30.7M 1s 2024-09-17T23:32:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138650K .......... .......... .......... .......... .......... 82% 24.0M 1s 2024-09-17T23:32:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138700K .......... .......... .......... .......... .......... 82% 51.7M 1s 2024-09-17T23:32:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138750K .......... .......... .......... .......... .......... 82% 16.8M 1s 2024-09-17T23:32:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138800K .......... .......... .......... .......... .......... 82% 42.9M 1s 2024-09-17T23:32:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138850K .......... .......... .......... .......... .......... 82% 21.3M 1s 2024-09-17T23:32:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138900K .......... .......... .......... .......... .......... 82% 38.7M 1s 2024-09-17T23:32:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:138950K .......... .......... .......... .......... .......... 82% 25.2M 1s 2024-09-17T23:32:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139000K .......... .......... .......... .......... .......... 82% 47.1M 1s 2024-09-17T23:32:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139050K .......... .......... .......... .......... .......... 82% 23.2M 1s 2024-09-17T23:32:52.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139100K .......... .......... .......... .......... .......... 82% 46.2M 1s 2024-09-17T23:32:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139150K .......... .......... .......... .......... .......... 82% 23.3M 1s 2024-09-17T23:32:52.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139200K .......... .......... .......... .......... .......... 82% 24.5M 1s 2024-09-17T23:32:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139250K .......... .......... .......... .......... .......... 82% 43.0M 1s 2024-09-17T23:32:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139300K .......... .......... .......... .......... .......... 82% 34.2M 1s 2024-09-17T23:32:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139350K .......... .......... .......... .......... .......... 82% 25.4M 1s 2024-09-17T23:32:52.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139400K .......... .......... .......... .......... .......... 82% 23.3M 1s 2024-09-17T23:32:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139450K .......... .......... .......... .......... .......... 82% 25.9M 1s 2024-09-17T23:32:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139500K .......... .......... .......... .......... .......... 82% 46.0M 1s 2024-09-17T23:32:52.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139550K .......... .......... .......... .......... .......... 82% 26.2M 1s 2024-09-17T23:32:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139600K .......... .......... .......... .......... .......... 82% 23.0M 1s 2024-09-17T23:32:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139650K .......... .......... .......... .......... .......... 82% 39.5M 1s 2024-09-17T23:32:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139700K .......... .......... .......... .......... .......... 82% 23.5M 1s 2024-09-17T23:32:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139750K .......... .......... .......... .......... .......... 82% 58.4M 1s 2024-09-17T23:32:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139800K .......... .......... .......... .......... .......... 82% 21.9M 1s 2024-09-17T23:32:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139850K .......... .......... .......... .......... .......... 82% 37.9M 1s 2024-09-17T23:32:52.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139900K .......... .......... .......... .......... .......... 83% 26.0M 1s 2024-09-17T23:32:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:139950K .......... .......... .......... .......... .......... 83% 21.9M 1s 2024-09-17T23:32:52.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140000K .......... .......... .......... .......... .......... 83% 24.6M 1s 2024-09-17T23:32:52.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140050K .......... .......... .......... .......... .......... 83% 52.9M 1s 2024-09-17T23:32:52.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140100K .......... .......... .......... .......... .......... 83% 23.6M 1s 2024-09-17T23:32:52.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140150K .......... .......... .......... .......... .......... 83% 41.7M 1s 2024-09-17T23:32:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140200K .......... .......... .......... .......... .......... 83% 24.1M 1s 2024-09-17T23:32:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140250K .......... .......... .......... .......... .......... 83% 48.8M 1s 2024-09-17T23:32:52.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140300K .......... .......... .......... .......... .......... 83% 21.5M 1s 2024-09-17T23:32:52.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140350K .......... .......... .......... .......... .......... 83% 23.5M 1s 2024-09-17T23:32:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140400K .......... .......... .......... .......... .......... 83% 37.5M 1s 2024-09-17T23:32:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140450K .......... .......... .......... .......... .......... 83% 23.5M 1s 2024-09-17T23:32:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140500K .......... .......... .......... .......... .......... 83% 23.7M 1s 2024-09-17T23:32:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140550K .......... .......... .......... .......... .......... 83% 54.8M 1s 2024-09-17T23:32:52.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140600K .......... .......... .......... .......... .......... 83% 22.8M 1s 2024-09-17T23:32:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140650K .......... .......... .......... .......... .......... 83% 38.1M 1s 2024-09-17T23:32:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140700K .......... .......... .......... .......... .......... 83% 24.5M 1s 2024-09-17T23:32:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140750K .......... .......... .......... .......... .......... 83% 22.6M 1s 2024-09-17T23:32:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140800K .......... .......... .......... .......... .......... 83% 44.4M 1s 2024-09-17T23:32:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140850K .......... .......... .......... .......... .......... 83% 24.8M 1s 2024-09-17T23:32:52.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140900K .......... .......... .......... .......... .......... 83% 43.8M 1s 2024-09-17T23:32:52.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:140950K .......... .......... .......... .......... .......... 83% 24.1M 1s 2024-09-17T23:32:52.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141000K .......... .......... .......... .......... .......... 83% 52.3M 1s 2024-09-17T23:32:52.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141050K .......... .......... .......... .......... .......... 83% 22.6M 1s 2024-09-17T23:32:52.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141100K .......... .......... .......... .......... .......... 83% 22.6M 1s 2024-09-17T23:32:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141150K .......... .......... .......... .......... .......... 83% 40.3M 1s 2024-09-17T23:32:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141200K .......... .......... .......... .......... .......... 83% 24.8M 1s 2024-09-17T23:32:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141250K .......... .......... .......... .......... .......... 83% 23.2M 1s 2024-09-17T23:32:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141300K .......... .......... .......... .......... .......... 83% 62.1M 1s 2024-09-17T23:32:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141350K .......... .......... .......... .......... .......... 83% 22.0M 1s 2024-09-17T23:32:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141400K .......... .......... .......... .......... .......... 83% 40.3M 1s 2024-09-17T23:32:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141450K .......... .......... .......... .......... .......... 83% 23.5M 1s 2024-09-17T23:32:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141500K .......... .......... .......... .......... .......... 83% 53.3M 1s 2024-09-17T23:32:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141550K .......... .......... .......... .......... .......... 84% 23.7M 1s 2024-09-17T23:32:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141600K .......... .......... .......... .......... .......... 84% 23.8M 1s 2024-09-17T23:32:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141650K .......... .......... .......... .......... .......... 84% 37.1M 1s 2024-09-17T23:32:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141700K .......... .......... .......... .......... .......... 84% 23.9M 1s 2024-09-17T23:32:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141750K .......... .......... .......... .......... .......... 84% 22.5M 1s 2024-09-17T23:32:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141800K .......... .......... .......... .......... .......... 84% 78.5M 1s 2024-09-17T23:32:52.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141850K .......... .......... .......... .......... .......... 84% 21.2M 1s 2024-09-17T23:32:52.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141900K .......... .......... .......... .......... .......... 84% 41.9M 1s 2024-09-17T23:32:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:141950K .......... .......... .......... .......... .......... 84% 22.2M 1s 2024-09-17T23:32:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142000K .......... .......... .......... .......... .......... 84% 25.4M 1s 2024-09-17T23:32:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142050K .......... .......... .......... .......... .......... 84% 52.3M 1s 2024-09-17T23:32:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142100K .......... .......... .......... .......... .......... 84% 22.8M 1s 2024-09-17T23:32:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142150K .......... .......... .......... .......... .......... 84% 42.8M 1s 2024-09-17T23:32:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142200K .......... .......... .......... .......... .......... 84% 23.1M 1s 2024-09-17T23:32:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142250K .......... .......... .......... .......... .......... 84% 24.0M 1s 2024-09-17T23:32:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142300K .......... .......... .......... .......... .......... 84% 52.2M 1s 2024-09-17T23:32:52.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142350K .......... .......... .......... .......... .......... 84% 23.3M 1s 2024-09-17T23:32:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142400K .......... .......... .......... .......... .......... 84% 22.8M 1s 2024-09-17T23:32:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142450K .......... .......... .......... .......... .......... 84% 37.3M 1s 2024-09-17T23:32:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142500K .......... .......... .......... .......... .......... 84% 24.3M 1s 2024-09-17T23:32:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142550K .......... .......... .......... .......... .......... 84% 46.9M 1s 2024-09-17T23:32:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142600K .......... .......... .......... .......... .......... 84% 26.7M 1s 2024-09-17T23:32:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142650K .......... .......... .......... .......... .......... 84% 42.7M 1s 2024-09-17T23:32:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142700K .......... .......... .......... .......... .......... 84% 22.2M 1s 2024-09-17T23:32:52.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142750K .......... .......... .......... .......... .......... 84% 22.7M 1s 2024-09-17T23:32:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142800K .......... .......... .......... .......... .......... 84% 25.6M 1s 2024-09-17T23:32:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142850K .......... .......... .......... .......... .......... 84% 47.3M 1s 2024-09-17T23:32:52.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142900K .......... .......... .......... .......... .......... 84% 24.3M 1s 2024-09-17T23:32:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:142950K .......... .......... .......... .......... .......... 84% 41.3M 1s 2024-09-17T23:32:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143000K .......... .......... .......... .......... .......... 84% 23.4M 1s 2024-09-17T23:32:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143050K .......... .......... .......... .......... .......... 84% 50.9M 1s 2024-09-17T23:32:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143100K .......... .......... .......... .......... .......... 84% 24.3M 1s 2024-09-17T23:32:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143150K .......... .......... .......... .......... .......... 84% 23.0M 1s 2024-09-17T23:32:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143200K .......... .......... .......... .......... .......... 84% 40.9M 1s 2024-09-17T23:32:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143250K .......... .......... .......... .......... .......... 85% 22.3M 1s 2024-09-17T23:32:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143300K .......... .......... .......... .......... .......... 85% 25.3M 1s 2024-09-17T23:32:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143350K .......... .......... .......... .......... .......... 85% 43.5M 1s 2024-09-17T23:32:52.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143400K .......... .......... .......... .......... .......... 85% 24.5M 1s 2024-09-17T23:32:52.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143450K .......... .......... .......... .......... .......... 85% 42.5M 1s 2024-09-17T23:32:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143500K .......... .......... .......... .......... .......... 85% 23.6M 1s 2024-09-17T23:32:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143550K .......... .......... .......... .......... .......... 85% 24.0M 1s 2024-09-17T23:32:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143600K .......... .......... .......... .......... .......... 85% 49.0M 1s 2024-09-17T23:32:52.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143650K .......... .......... .......... .......... .......... 85% 23.8M 1s 2024-09-17T23:32:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143700K .......... .......... .......... .......... .......... 85% 40.3M 1s 2024-09-17T23:32:52.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143750K .......... .......... .......... .......... .......... 85% 22.8M 1s 2024-09-17T23:32:52.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143800K .......... .......... .......... .......... .......... 85% 52.3M 1s 2024-09-17T23:32:52.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143850K .......... .......... .......... .......... .......... 85% 24.1M 1s 2024-09-17T23:32:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143900K .......... .......... .......... .......... .......... 85% 39.2M 1s 2024-09-17T23:32:52.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:143950K .......... .......... .......... .......... .......... 85% 17.0M 1s 2024-09-17T23:32:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144000K .......... .......... .......... .......... .......... 85% 40.0M 1s 2024-09-17T23:32:52.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144050K .......... .......... .......... .......... .......... 85% 22.8M 1s 2024-09-17T23:32:52.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144100K .......... .......... .......... .......... .......... 85% 51.5M 1s 2024-09-17T23:32:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144150K .......... .......... .......... .......... .......... 85% 23.7M 1s 2024-09-17T23:32:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144200K .......... .......... .......... .......... .......... 85% 42.6M 1s 2024-09-17T23:32:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144250K .......... .......... .......... .......... .......... 85% 22.0M 1s 2024-09-17T23:32:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144300K .......... .......... .......... .......... .......... 85% 59.8M 1s 2024-09-17T23:32:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144350K .......... .......... .......... .......... .......... 85% 22.7M 1s 2024-09-17T23:32:52.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144400K .......... .......... .......... .......... .......... 85% 23.8M 1s 2024-09-17T23:32:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144450K .......... .......... .......... .......... .......... 85% 39.3M 1s 2024-09-17T23:32:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144500K .......... .......... .......... .......... .......... 85% 22.7M 1s 2024-09-17T23:32:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144550K .......... .......... .......... .......... .......... 85% 50.7M 1s 2024-09-17T23:32:52.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144600K .......... .......... .......... .......... .......... 85% 25.6M 1s 2024-09-17T23:32:52.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144650K .......... .......... .......... .......... .......... 85% 31.4M 1s 2024-09-17T23:32:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144700K .......... .......... .......... .......... .......... 85% 11.2M 1s 2024-09-17T23:32:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144750K .......... .......... .......... .......... .......... 85% 215M 1s 2024-09-17T23:32:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144800K .......... .......... .......... .......... .......... 85% 36.5M 1s 2024-09-17T23:32:52.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144850K .......... .......... .......... .......... .......... 85% 53.8M 1s 2024-09-17T23:32:52.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144900K .......... .......... .......... .......... .......... 85% 24.3M 1s 2024-09-17T23:32:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:144950K .......... .......... .......... .......... .......... 86% 21.9M 1s 2024-09-17T23:32:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145000K .......... .......... .......... .......... .......... 86% 72.3M 1s 2024-09-17T23:32:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145050K .......... .......... .......... .......... .......... 86% 18.6M 1s 2024-09-17T23:32:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145100K .......... .......... .......... .......... .......... 86% 42.2M 1s 2024-09-17T23:32:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145150K .......... .......... .......... .......... .......... 86% 26.5M 1s 2024-09-17T23:32:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145200K .......... .......... .......... .......... .......... 86% 22.2M 1s 2024-09-17T23:32:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145250K .......... .......... .......... .......... .......... 86% 32.2M 1s 2024-09-17T23:32:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145300K .......... .......... .......... .......... .......... 86% 26.7M 1s 2024-09-17T23:32:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145350K .......... .......... .......... .......... .......... 86% 55.1M 1s 2024-09-17T23:32:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145400K .......... .......... .......... .......... .......... 86% 33.3M 1s 2024-09-17T23:32:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145450K .......... .......... .......... .......... .......... 86% 27.9M 1s 2024-09-17T23:32:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145500K .......... .......... .......... .......... .......... 86% 23.8M 1s 2024-09-17T23:32:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145550K .......... .......... .......... .......... .......... 86% 24.9M 1s 2024-09-17T23:32:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145600K .......... .......... .......... .......... .......... 86% 22.4M 1s 2024-09-17T23:32:52.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145650K .......... .......... .......... .......... .......... 86% 51.8M 1s 2024-09-17T23:32:52.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145700K .......... .......... .......... .......... .......... 86% 30.2M 1s 2024-09-17T23:32:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145750K .......... .......... .......... .......... .......... 86% 25.2M 1s 2024-09-17T23:32:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145800K .......... .......... .......... .......... .......... 86% 59.7M 1s 2024-09-17T23:32:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145850K .......... .......... .......... .......... .......... 86% 25.4M 1s 2024-09-17T23:32:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145900K .......... .......... .......... .......... .......... 86% 26.1M 1s 2024-09-17T23:32:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:145950K .......... .......... .......... .......... .......... 86% 21.8M 1s 2024-09-17T23:32:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146000K .......... .......... .......... .......... .......... 86% 32.5M 1s 2024-09-17T23:32:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146050K .......... .......... .......... .......... .......... 86% 28.8M 1s 2024-09-17T23:32:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146100K .......... .......... .......... .......... .......... 86% 44.3M 1s 2024-09-17T23:32:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146150K .......... .......... .......... .......... .......... 86% 23.9M 1s 2024-09-17T23:32:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146200K .......... .......... .......... .......... .......... 86% 33.7M 1s 2024-09-17T23:32:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146250K .......... .......... .......... .......... .......... 86% 27.5M 1s 2024-09-17T23:32:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146300K .......... .......... .......... .......... .......... 86% 46.1M 1s 2024-09-17T23:32:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146350K .......... .......... .......... .......... .......... 86% 15.9M 1s 2024-09-17T23:32:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146400K .......... .......... .......... .......... .......... 86% 51.6M 1s 2024-09-17T23:32:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146450K .......... .......... .......... .......... .......... 86% 28.8M 1s 2024-09-17T23:32:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146500K .......... .......... .......... .......... .......... 86% 30.3M 1s 2024-09-17T23:32:52.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146550K .......... .......... .......... .......... .......... 86% 21.5M 1s 2024-09-17T23:32:52.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146600K .......... .......... .......... .......... .......... 86% 67.5M 1s 2024-09-17T23:32:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146650K .......... .......... .......... .......... .......... 87% 24.1M 1s 2024-09-17T23:32:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146700K .......... .......... .......... .......... .......... 87% 32.8M 1s 2024-09-17T23:32:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146750K .......... .......... .......... .......... .......... 87% 19.8M 1s 2024-09-17T23:32:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146800K .......... .......... .......... .......... .......... 87% 34.6M 1s 2024-09-17T23:32:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146850K .......... .......... .......... .......... .......... 87% 36.0M 1s 2024-09-17T23:32:52.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146900K .......... .......... .......... .......... .......... 87% 30.1M 1s 2024-09-17T23:32:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:146950K .......... .......... .......... .......... .......... 87% 29.4M 1s 2024-09-17T23:32:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147000K .......... .......... .......... .......... .......... 87% 26.9M 1s 2024-09-17T23:32:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147050K .......... .......... .......... .......... .......... 87% 26.0M 1s 2024-09-17T23:32:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147100K .......... .......... .......... .......... .......... 87% 50.0M 1s 2024-09-17T23:32:52.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147150K .......... .......... .......... .......... .......... 87% 23.1M 1s 2024-09-17T23:32:52.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147200K .......... .......... .......... .......... .......... 87% 23.5M 1s 2024-09-17T23:32:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147250K .......... .......... .......... .......... .......... 87% 32.4M 1s 2024-09-17T23:32:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147300K .......... .......... .......... .......... .......... 87% 28.5M 1s 2024-09-17T23:32:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147350K .......... .......... .......... .......... .......... 87% 22.9M 1s 2024-09-17T23:32:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147400K .......... .......... .......... .......... .......... 87% 52.4M 1s 2024-09-17T23:32:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147450K .......... .......... .......... .......... .......... 87% 24.6M 1s 2024-09-17T23:32:52.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147500K .......... .......... .......... .......... .......... 87% 39.2M 1s 2024-09-17T23:32:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147550K .......... .......... .......... .......... .......... 87% 23.9M 1s 2024-09-17T23:32:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147600K .......... .......... .......... .......... .......... 87% 38.1M 1s 2024-09-17T23:32:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147650K .......... .......... .......... .......... .......... 87% 28.2M 1s 2024-09-17T23:32:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147700K .......... .......... .......... .......... .......... 87% 23.4M 1s 2024-09-17T23:32:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147750K .......... .......... .......... .......... .......... 87% 35.2M 1s 2024-09-17T23:32:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147800K .......... .......... .......... .......... .......... 87% 25.0M 1s 2024-09-17T23:32:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147850K .......... .......... .......... .......... .......... 87% 22.4M 1s 2024-09-17T23:32:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147900K .......... .......... .......... .......... .......... 87% 57.1M 1s 2024-09-17T23:32:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:147950K .......... .......... .......... .......... .......... 87% 22.9M 1s 2024-09-17T23:32:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148000K .......... .......... .......... .......... .......... 87% 22.9M 1s 2024-09-17T23:32:52.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148050K .......... .......... .......... .......... .......... 87% 44.9M 1s 2024-09-17T23:32:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148100K .......... .......... .......... .......... .......... 87% 22.6M 1s 2024-09-17T23:32:52.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148150K .......... .......... .......... .......... .......... 87% 55.8M 1s 2024-09-17T23:32:52.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148200K .......... .......... .......... .......... .......... 87% 23.8M 1s 2024-09-17T23:32:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148250K .......... .......... .......... .......... .......... 87% 27.3M 1s 2024-09-17T23:32:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148300K .......... .......... .......... .......... .......... 88% 46.6M 1s 2024-09-17T23:32:52.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148350K .......... .......... .......... .......... .......... 88% 19.4M 1s 2024-09-17T23:32:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148400K .......... .......... .......... .......... .......... 88% 22.2M 1s 2024-09-17T23:32:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148450K .......... .......... .......... .......... .......... 88% 68.7M 1s 2024-09-17T23:32:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148500K .......... .......... .......... .......... .......... 88% 22.5M 1s 2024-09-17T23:32:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148550K .......... .......... .......... .......... .......... 88% 25.3M 1s 2024-09-17T23:32:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148600K .......... .......... .......... .......... .......... 88% 41.0M 1s 2024-09-17T23:32:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148650K .......... .......... .......... .......... .......... 88% 49.2M 1s 2024-09-17T23:32:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148700K .......... .......... .......... .......... .......... 88% 24.3M 1s 2024-09-17T23:32:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148750K .......... .......... .......... .......... .......... 88% 21.3M 1s 2024-09-17T23:32:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148800K .......... .......... .......... .......... .......... 88% 27.6M 1s 2024-09-17T23:32:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148850K .......... .......... .......... .......... .......... 88% 33.1M 1s 2024-09-17T23:32:52.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148900K .......... .......... .......... .......... .......... 88% 50.6M 1s 2024-09-17T23:32:52.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:148950K .......... .......... .......... .......... .......... 88% 23.7M 1s 2024-09-17T23:32:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149000K .......... .......... .......... .......... .......... 88% 26.2M 1s 2024-09-17T23:32:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149050K .......... .......... .......... .......... .......... 88% 33.2M 1s 2024-09-17T23:32:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149100K .......... .......... .......... .......... .......... 88% 36.7M 1s 2024-09-17T23:32:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149150K .......... .......... .......... .......... .......... 88% 17.3M 1s 2024-09-17T23:32:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149200K .......... .......... .......... .......... .......... 88% 51.8M 1s 2024-09-17T23:32:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149250K .......... .......... .......... .......... .......... 88% 26.5M 1s 2024-09-17T23:32:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149300K .......... .......... .......... .......... .......... 88% 21.7M 1s 2024-09-17T23:32:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149350K .......... .......... .......... .......... .......... 88% 77.2M 1s 2024-09-17T23:32:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149400K .......... .......... .......... .......... .......... 88% 32.5M 1s 2024-09-17T23:32:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149450K .......... .......... .......... .......... .......... 88% 22.9M 1s 2024-09-17T23:32:52.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149500K .......... .......... .......... .......... .......... 88% 39.8M 1s 2024-09-17T23:32:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149550K .......... .......... .......... .......... .......... 88% 22.0M 1s 2024-09-17T23:32:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149600K .......... .......... .......... .......... .......... 88% 24.1M 1s 2024-09-17T23:32:52.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149650K .......... .......... .......... .......... .......... 88% 23.2M 1s 2024-09-17T23:32:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149700K .......... .......... .......... .......... .......... 88% 54.4M 1s 2024-09-17T23:32:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149750K .......... .......... .......... .......... .......... 88% 23.9M 1s 2024-09-17T23:32:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149800K .......... .......... .......... .......... .......... 88% 28.5M 1s 2024-09-17T23:32:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149850K .......... .......... .......... .......... .......... 88% 32.1M 1s 2024-09-17T23:32:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149900K .......... .......... .......... .......... .......... 88% 45.7M 1s 2024-09-17T23:32:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:149950K .......... .......... .......... .......... .......... 88% 24.5M 1s 2024-09-17T23:32:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150000K .......... .......... .......... .......... .......... 89% 22.9M 1s 2024-09-17T23:32:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150050K .......... .......... .......... .......... .......... 89% 36.3M 1s 2024-09-17T23:32:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150100K .......... .......... .......... .......... .......... 89% 24.2M 1s 2024-09-17T23:32:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150150K .......... .......... .......... .......... .......... 89% 24.4M 1s 2024-09-17T23:32:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150200K .......... .......... .......... .......... .......... 89% 51.0M 1s 2024-09-17T23:32:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150250K .......... .......... .......... .......... .......... 89% 23.6M 1s 2024-09-17T23:32:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150300K .......... .......... .......... .......... .......... 89% 40.9M 1s 2024-09-17T23:32:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150350K .......... .......... .......... .......... .......... 89% 27.6M 1s 2024-09-17T23:32:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150400K .......... .......... .......... .......... .......... 89% 21.1M 1s 2024-09-17T23:32:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150450K .......... .......... .......... .......... .......... 89% 47.7M 1s 2024-09-17T23:32:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150500K .......... .......... .......... .......... .......... 89% 28.0M 1s 2024-09-17T23:32:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150550K .......... .......... .......... .......... .......... 89% 28.2M 1s 2024-09-17T23:32:52.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150600K .......... .......... .......... .......... .......... 89% 25.6M 1s 2024-09-17T23:32:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150650K .......... .......... .......... .......... .......... 89% 33.1M 1s 2024-09-17T23:32:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150700K .......... .......... .......... .......... .......... 89% 34.2M 1s 2024-09-17T23:32:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150750K .......... .......... .......... .......... .......... 89% 22.9M 1s 2024-09-17T23:32:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150800K .......... .......... .......... .......... .......... 89% 23.8M 1s 2024-09-17T23:32:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150850K .......... .......... .......... .......... .......... 89% 38.4M 1s 2024-09-17T23:32:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150900K .......... .......... .......... .......... .......... 89% 33.0M 1s 2024-09-17T23:32:52.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:150950K .......... .......... .......... .......... .......... 89% 29.4M 1s 2024-09-17T23:32:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151000K .......... .......... .......... .......... .......... 89% 24.8M 1s 2024-09-17T23:32:52.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151050K .......... .......... .......... .......... .......... 89% 24.3M 1s 2024-09-17T23:32:52.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151100K .......... .......... .......... .......... .......... 89% 40.1M 1s 2024-09-17T23:32:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151150K .......... .......... .......... .......... .......... 89% 28.8M 1s 2024-09-17T23:32:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151200K .......... .......... .......... .......... .......... 89% 18.4M 1s 2024-09-17T23:32:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151250K .......... .......... .......... .......... .......... 89% 116M 1s 2024-09-17T23:32:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151300K .......... .......... .......... .......... .......... 89% 26.0M 1s 2024-09-17T23:32:52.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151350K .......... .......... .......... .......... .......... 89% 26.6M 1s 2024-09-17T23:32:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151400K .......... .......... .......... .......... .......... 89% 23.9M 1s 2024-09-17T23:32:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151450K .......... .......... .......... .......... .......... 89% 52.5M 1s 2024-09-17T23:32:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151500K .......... .......... .......... .......... .......... 89% 24.5M 1s 2024-09-17T23:32:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151550K .......... .......... .......... .......... .......... 89% 33.4M 1s 2024-09-17T23:32:52.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151600K .......... .......... .......... .......... .......... 89% 25.8M 1s 2024-09-17T23:32:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151650K .......... .......... .......... .......... .......... 89% 22.4M 1s 2024-09-17T23:32:52.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151700K .......... .......... .......... .......... .......... 90% 58.2M 1s 2024-09-17T23:32:52.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151750K .......... .......... .......... .......... .......... 90% 21.2M 1s 2024-09-17T23:32:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151800K .......... .......... .......... .......... .......... 90% 25.2M 1s 2024-09-17T23:32:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151850K .......... .......... .......... .......... .......... 90% 61.5M 1s 2024-09-17T23:32:52.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151900K .......... .......... .......... .......... .......... 90% 24.7M 1s 2024-09-17T23:32:52.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:151950K .......... .......... .......... .......... .......... 90% 19.9M 1s 2024-09-17T23:32:52.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152000K .......... .......... .......... .......... .......... 90% 49.5M 1s 2024-09-17T23:32:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152050K .......... .......... .......... .......... .......... 90% 24.0M 1s 2024-09-17T23:32:52.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152100K .......... .......... .......... .......... .......... 90% 40.5M 1s 2024-09-17T23:32:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152150K .......... .......... .......... .......... .......... 90% 25.0M 1s 2024-09-17T23:32:52.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152200K .......... .......... .......... .......... .......... 90% 43.7M 1s 2024-09-17T23:32:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152250K .......... .......... .......... .......... .......... 90% 23.5M 1s 2024-09-17T23:32:52.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152300K .......... .......... .......... .......... .......... 90% 34.6M 1s 2024-09-17T23:32:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152350K .......... .......... .......... .......... .......... 90% 24.4M 1s 2024-09-17T23:32:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152400K .......... .......... .......... .......... .......... 90% 23.3M 1s 2024-09-17T23:32:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152450K .......... .......... .......... .......... .......... 90% 29.5M 1s 2024-09-17T23:32:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152500K .......... .......... .......... .......... .......... 90% 40.8M 1s 2024-09-17T23:32:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152550K .......... .......... .......... .......... .......... 90% 23.7M 1s 2024-09-17T23:32:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152600K .......... .......... .......... .......... .......... 90% 38.1M 1s 2024-09-17T23:32:52.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152650K .......... .......... .......... .......... .......... 90% 25.2M 1s 2024-09-17T23:32:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152700K .......... .......... .......... .......... .......... 90% 54.6M 1s 2024-09-17T23:32:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152750K .......... .......... .......... .......... .......... 90% 22.2M 1s 2024-09-17T23:32:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152800K .......... .......... .......... .......... .......... 90% 34.8M 1s 2024-09-17T23:32:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152850K .......... .......... .......... .......... .......... 90% 17.8M 1s 2024-09-17T23:32:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152900K .......... .......... .......... .......... .......... 90% 43.7M 1s 2024-09-17T23:32:52.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:152950K .......... .......... .......... .......... .......... 90% 24.8M 1s 2024-09-17T23:32:52.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153000K .......... .......... .......... .......... .......... 90% 42.8M 1s 2024-09-17T23:32:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153050K .......... .......... .......... .......... .......... 90% 24.5M 1s 2024-09-17T23:32:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153100K .......... .......... .......... .......... .......... 90% 39.0M 1s 2024-09-17T23:32:52.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153150K .......... .......... .......... .......... .......... 90% 21.2M 1s 2024-09-17T23:32:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153200K .......... .......... .......... .......... .......... 90% 26.1M 1s 2024-09-17T23:32:52.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153250K .......... .......... .......... .......... .......... 90% 47.7M 1s 2024-09-17T23:32:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153300K .......... .......... .......... .......... .......... 90% 24.6M 1s 2024-09-17T23:32:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153350K .......... .......... .......... .......... .......... 91% 35.9M 1s 2024-09-17T23:32:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153400K .......... .......... .......... .......... .......... 91% 26.0M 1s 2024-09-17T23:32:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153450K .......... .......... .......... .......... .......... 91% 45.0M 1s 2024-09-17T23:32:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153500K .......... .......... .......... .......... .......... 91% 23.8M 1s 2024-09-17T23:32:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153550K .......... .......... .......... .......... .......... 91% 23.4M 1s 2024-09-17T23:32:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153600K .......... .......... .......... .......... .......... 91% 22.9M 1s 2024-09-17T23:32:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153650K .......... .......... .......... .......... .......... 91% 41.6M 1s 2024-09-17T23:32:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153700K .......... .......... .......... .......... .......... 91% 24.6M 1s 2024-09-17T23:32:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153750K .......... .......... .......... .......... .......... 91% 41.0M 1s 2024-09-17T23:32:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153800K .......... .......... .......... .......... .......... 91% 24.5M 1s 2024-09-17T23:32:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153850K .......... .......... .......... .......... .......... 91% 45.8M 0s 2024-09-17T23:32:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153900K .......... .......... .......... .......... .......... 91% 21.0M 0s 2024-09-17T23:32:52.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:153950K .......... .......... .......... .......... .......... 91% 24.6M 0s 2024-09-17T23:32:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154000K .......... .......... .......... .......... .......... 91% 52.0M 0s 2024-09-17T23:32:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154050K .......... .......... .......... .......... .......... 91% 23.3M 0s 2024-09-17T23:32:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154100K .......... .......... .......... .......... .......... 91% 38.4M 0s 2024-09-17T23:32:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154150K .......... .......... .......... .......... .......... 91% 22.5M 0s 2024-09-17T23:32:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154200K .......... .......... .......... .......... .......... 91% 24.4M 0s 2024-09-17T23:32:52.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154250K .......... .......... .......... .......... .......... 91% 49.2M 0s 2024-09-17T23:32:52.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154300K .......... .......... .......... .......... .......... 91% 24.9M 0s 2024-09-17T23:32:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154350K .......... .......... .......... .......... .......... 91% 23.6M 0s 2024-09-17T23:32:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154400K .......... .......... .......... .......... .......... 91% 45.3M 0s 2024-09-17T23:32:52.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154450K .......... .......... .......... .......... .......... 91% 24.6M 0s 2024-09-17T23:32:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154500K .......... .......... .......... .......... .......... 91% 40.7M 0s 2024-09-17T23:32:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154550K .......... .......... .......... .......... .......... 91% 25.0M 0s 2024-09-17T23:32:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154600K .......... .......... .......... .......... .......... 91% 40.9M 0s 2024-09-17T23:32:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154650K .......... .......... .......... .......... .......... 91% 23.2M 0s 2024-09-17T23:32:52.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154700K .......... .......... .......... .......... .......... 91% 24.6M 0s 2024-09-17T23:32:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154750K .......... .......... .......... .......... .......... 91% 25.3M 0s 2024-09-17T23:32:52.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154800K .......... .......... .......... .......... .......... 91% 36.6M 0s 2024-09-17T23:32:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154850K .......... .......... .......... .......... .......... 91% 26.1M 0s 2024-09-17T23:32:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154900K .......... .......... .......... .......... .......... 91% 36.8M 0s 2024-09-17T23:32:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:154950K .......... .......... .......... .......... .......... 91% 24.4M 0s 2024-09-17T23:32:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155000K .......... .......... .......... .......... .......... 91% 47.6M 0s 2024-09-17T23:32:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155050K .......... .......... .......... .......... .......... 92% 24.5M 0s 2024-09-17T23:32:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155100K .......... .......... .......... .......... .......... 92% 35.8M 0s 2024-09-17T23:32:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155150K .......... .......... .......... .......... .......... 92% 25.1M 0s 2024-09-17T23:32:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155200K .......... .......... .......... .......... .......... 92% 23.9M 0s 2024-09-17T23:32:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155250K .......... .......... .......... .......... .......... 92% 25.6M 0s 2024-09-17T23:32:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155300K .......... .......... .......... .......... .......... 92% 43.6M 0s 2024-09-17T23:32:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155350K .......... .......... .......... .......... .......... 92% 24.3M 0s 2024-09-17T23:32:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155400K .......... .......... .......... .......... .......... 92% 39.8M 0s 2024-09-17T23:32:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155450K .......... .......... .......... .......... .......... 92% 23.5M 0s 2024-09-17T23:32:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155500K .......... .......... .......... .......... .......... 92% 51.4M 0s 2024-09-17T23:32:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155550K .......... .......... .......... .......... .......... 92% 24.1M 0s 2024-09-17T23:32:52.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155600K .......... .......... .......... .......... .......... 92% 23.0M 0s 2024-09-17T23:32:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155650K .......... .......... .......... .......... .......... 92% 41.4M 0s 2024-09-17T23:32:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155700K .......... .......... .......... .......... .......... 92% 22.5M 0s 2024-09-17T23:32:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155750K .......... .......... .......... .......... .......... 92% 23.2M 0s 2024-09-17T23:32:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155800K .......... .......... .......... .......... .......... 92% 50.9M 0s 2024-09-17T23:32:52.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155850K .......... .......... .......... .......... .......... 92% 34.1M 0s 2024-09-17T23:32:52.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155900K .......... .......... .......... .......... .......... 92% 29.2M 0s 2024-09-17T23:32:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:155950K .......... .......... .......... .......... .......... 92% 21.6M 0s 2024-09-17T23:32:52.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156000K .......... .......... .......... .......... .......... 92% 24.2M 0s 2024-09-17T23:32:52.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156050K .......... .......... .......... .......... .......... 92% 48.9M 0s 2024-09-17T23:32:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156100K .......... .......... .......... .......... .......... 92% 26.7M 0s 2024-09-17T23:32:52.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156150K .......... .......... .......... .......... .......... 92% 41.3M 0s 2024-09-17T23:32:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156200K .......... .......... .......... .......... .......... 92% 22.8M 0s 2024-09-17T23:32:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156250K .......... .......... .......... .......... .......... 92% 49.7M 0s 2024-09-17T23:32:52.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156300K .......... .......... .......... .......... .......... 92% 22.6M 0s 2024-09-17T23:32:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156350K .......... .......... .......... .......... .......... 92% 23.8M 0s 2024-09-17T23:32:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156400K .......... .......... .......... .......... .......... 92% 36.8M 0s 2024-09-17T23:32:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156450K .......... .......... .......... .......... .......... 92% 23.5M 0s 2024-09-17T23:32:52.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156500K .......... .......... .......... .......... .......... 92% 24.9M 0s 2024-09-17T23:32:52.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156550K .......... .......... .......... .......... .......... 92% 55.7M 0s 2024-09-17T23:32:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156600K .......... .......... .......... .......... .......... 92% 23.8M 0s 2024-09-17T23:32:52.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156650K .......... .......... .......... .......... .......... 92% 41.1M 0s 2024-09-17T23:32:52.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156700K .......... .......... .......... .......... .......... 92% 23.7M 0s 2024-09-17T23:32:52.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156750K .......... .......... .......... .......... .......... 93% 23.5M 0s 2024-09-17T23:32:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156800K .......... .......... .......... .......... .......... 93% 52.1M 0s 2024-09-17T23:32:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156850K .......... .......... .......... .......... .......... 93% 23.2M 0s 2024-09-17T23:32:52.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156900K .......... .......... .......... .......... .......... 93% 33.3M 0s 2024-09-17T23:32:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:156950K .......... .......... .......... .......... .......... 93% 24.9M 0s 2024-09-17T23:32:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157000K .......... .......... .......... .......... .......... 93% 25.2M 0s 2024-09-17T23:32:52.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157050K .......... .......... .......... .......... .......... 93% 47.7M 0s 2024-09-17T23:32:52.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157100K .......... .......... .......... .......... .......... 93% 23.5M 0s 2024-09-17T23:32:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157150K .......... .......... .......... .......... .......... 93% 28.1M 0s 2024-09-17T23:32:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157200K .......... .......... .......... .......... .......... 93% 33.2M 0s 2024-09-17T23:32:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157250K .......... .......... .......... .......... .......... 93% 24.7M 0s 2024-09-17T23:32:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157300K .......... .......... .......... .......... .......... 93% 49.9M 0s 2024-09-17T23:32:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157350K .......... .......... .......... .......... .......... 93% 23.8M 0s 2024-09-17T23:32:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157400K .......... .......... .......... .......... .......... 93% 40.7M 0s 2024-09-17T23:32:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157450K .......... .......... .......... .......... .......... 93% 23.7M 0s 2024-09-17T23:32:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157500K .......... .......... .......... .......... .......... 93% 23.3M 0s 2024-09-17T23:32:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157550K .......... .......... .......... .......... .......... 93% 23.2M 0s 2024-09-17T23:32:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157600K .......... .......... .......... .......... .......... 93% 44.3M 0s 2024-09-17T23:32:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157650K .......... .......... .......... .......... .......... 93% 25.2M 0s 2024-09-17T23:32:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157700K .......... .......... .......... .......... .......... 93% 37.0M 0s 2024-09-17T23:32:52.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157750K .......... .......... .......... .......... .......... 93% 22.2M 0s 2024-09-17T23:32:52.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157800K .......... .......... .......... .......... .......... 93% 57.7M 0s 2024-09-17T23:32:52.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157850K .......... .......... .......... .......... .......... 93% 27.8M 0s 2024-09-17T23:32:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157900K .......... .......... .......... .......... .......... 93% 30.1M 0s 2024-09-17T23:32:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:157950K .......... .......... .......... .......... .......... 93% 25.8M 0s 2024-09-17T23:32:52.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158000K .......... .......... .......... .......... .......... 93% 23.4M 0s 2024-09-17T23:32:53.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158050K .......... .......... .......... .......... .......... 93% 22.8M 0s 2024-09-17T23:32:53.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158100K .......... .......... .......... .......... .......... 93% 52.5M 0s 2024-09-17T23:32:53.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158150K .......... .......... .......... .......... .......... 93% 23.2M 0s 2024-09-17T23:32:53.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158200K .......... .......... .......... .......... .......... 93% 38.8M 0s 2024-09-17T23:32:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158250K .......... .......... .......... .......... .......... 93% 21.6M 0s 2024-09-17T23:32:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158300K .......... .......... .......... .......... .......... 93% 59.3M 0s 2024-09-17T23:32:53.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158350K .......... .......... .......... .......... .......... 93% 22.9M 0s 2024-09-17T23:32:53.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158400K .......... .......... .......... .......... .......... 93% 29.8M 0s 2024-09-17T23:32:53.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158450K .......... .......... .......... .......... .......... 94% 36.1M 0s 2024-09-17T23:32:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158500K .......... .......... .......... .......... .......... 94% 22.8M 0s 2024-09-17T23:32:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158550K .......... .......... .......... .......... .......... 94% 49.6M 0s 2024-09-17T23:32:53.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158600K .......... .......... .......... .......... .......... 94% 24.4M 0s 2024-09-17T23:32:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158650K .......... .......... .......... .......... .......... 94% 36.7M 0s 2024-09-17T23:32:53.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158700K .......... .......... .......... .......... .......... 94% 23.7M 0s 2024-09-17T23:32:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158750K .......... .......... .......... .......... .......... 94% 22.6M 0s 2024-09-17T23:32:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158800K .......... .......... .......... .......... .......... 94% 25.8M 0s 2024-09-17T23:32:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158850K .......... .......... .......... .......... .......... 94% 50.5M 0s 2024-09-17T23:32:53.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158900K .......... .......... .......... .......... .......... 94% 34.4M 0s 2024-09-17T23:32:53.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:158950K .......... .......... .......... .......... .......... 94% 24.4M 0s 2024-09-17T23:32:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159000K .......... .......... .......... .......... .......... 94% 25.4M 0s 2024-09-17T23:32:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159050K .......... .......... .......... .......... .......... 94% 52.6M 0s 2024-09-17T23:32:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159100K .......... .......... .......... .......... .......... 94% 23.5M 0s 2024-09-17T23:32:53.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159150K .......... .......... .......... .......... .......... 94% 24.2M 0s 2024-09-17T23:32:53.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159200K .......... .......... .......... .......... .......... 94% 39.3M 0s 2024-09-17T23:32:53.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159250K .......... .......... .......... .......... .......... 94% 23.6M 0s 2024-09-17T23:32:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159300K .......... .......... .......... .......... .......... 94% 23.3M 0s 2024-09-17T23:32:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159350K .......... .......... .......... .......... .......... 94% 53.0M 0s 2024-09-17T23:32:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159400K .......... .......... .......... .......... .......... 94% 34.6M 0s 2024-09-17T23:32:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159450K .......... .......... .......... .......... .......... 94% 26.4M 0s 2024-09-17T23:32:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159500K .......... .......... .......... .......... .......... 94% 23.8M 0s 2024-09-17T23:32:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159550K .......... .......... .......... .......... .......... 94% 22.7M 0s 2024-09-17T23:32:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159600K .......... .......... .......... .......... .......... 94% 57.1M 0s 2024-09-17T23:32:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159650K .......... .......... .......... .......... .......... 94% 23.9M 0s 2024-09-17T23:32:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159700K .......... .......... .......... .......... .......... 94% 39.5M 0s 2024-09-17T23:32:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159750K .......... .......... .......... .......... .......... 94% 20.8M 0s 2024-09-17T23:32:53.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159800K .......... .......... .......... .......... .......... 94% 26.7M 0s 2024-09-17T23:32:53.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159850K .......... .......... .......... .......... .......... 94% 49.4M 0s 2024-09-17T23:32:53.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159900K .......... .......... .......... .......... .......... 94% 23.7M 0s 2024-09-17T23:32:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:159950K .......... .......... .......... .......... .......... 94% 24.6M 0s 2024-09-17T23:32:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160000K .......... .......... .......... .......... .......... 94% 38.8M 0s 2024-09-17T23:32:53.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160050K .......... .......... .......... .......... .......... 94% 24.3M 0s 2024-09-17T23:32:53.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160100K .......... .......... .......... .......... .......... 95% 45.5M 0s 2024-09-17T23:32:53.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160150K .......... .......... .......... .......... .......... 95% 21.5M 0s 2024-09-17T23:32:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160200K .......... .......... .......... .......... .......... 95% 44.2M 0s 2024-09-17T23:32:53.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160250K .......... .......... .......... .......... .......... 95% 25.4M 0s 2024-09-17T23:32:53.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160300K .......... .......... .......... .......... .......... 95% 23.2M 0s 2024-09-17T23:32:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160350K .......... .......... .......... .......... .......... 95% 23.4M 0s 2024-09-17T23:32:53.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160400K .......... .......... .......... .......... .......... 95% 50.9M 0s 2024-09-17T23:32:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160450K .......... .......... .......... .......... .......... 95% 24.4M 0s 2024-09-17T23:32:53.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160500K .......... .......... .......... .......... .......... 95% 33.5M 0s 2024-09-17T23:32:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160550K .......... .......... .......... .......... .......... 95% 27.2M 0s 2024-09-17T23:32:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160600K .......... .......... .......... .......... .......... 95% 47.0M 0s 2024-09-17T23:32:53.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160650K .......... .......... .......... .......... .......... 95% 23.0M 0s 2024-09-17T23:32:53.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160700K .......... .......... .......... .......... .......... 95% 45.5M 0s 2024-09-17T23:32:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160750K .......... .......... .......... .......... .......... 95% 17.0M 0s 2024-09-17T23:32:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160800K .......... .......... .......... .......... .......... 95% 39.4M 0s 2024-09-17T23:32:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160850K .......... .......... .......... .......... .......... 95% 23.2M 0s 2024-09-17T23:32:53.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160900K .......... .......... .......... .......... .......... 95% 53.5M 0s 2024-09-17T23:32:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:160950K .......... .......... .......... .......... .......... 95% 22.8M 0s 2024-09-17T23:32:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161000K .......... .......... .......... .......... .......... 95% 37.1M 0s 2024-09-17T23:32:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161050K .......... .......... .......... .......... .......... 95% 24.4M 0s 2024-09-17T23:32:53.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161100K .......... .......... .......... .......... .......... 95% 56.0M 0s 2024-09-17T23:32:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161150K .......... .......... .......... .......... .......... 95% 23.4M 0s 2024-09-17T23:32:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161200K .......... .......... .......... .......... .......... 95% 23.6M 0s 2024-09-17T23:32:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161250K .......... .......... .......... .......... .......... 95% 40.1M 0s 2024-09-17T23:32:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161300K .......... .......... .......... .......... .......... 95% 23.1M 0s 2024-09-17T23:32:53.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161350K .......... .......... .......... .......... .......... 95% 53.0M 0s 2024-09-17T23:32:53.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161400K .......... .......... .......... .......... .......... 95% 27.2M 0s 2024-09-17T23:32:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161450K .......... .......... .......... .......... .......... 95% 29.7M 0s 2024-09-17T23:32:53.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161500K .......... .......... .......... .......... .......... 95% 24.6M 0s 2024-09-17T23:32:53.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161550K .......... .......... .......... .......... .......... 95% 24.2M 0s 2024-09-17T23:32:53.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161600K .......... .......... .......... .......... .......... 95% 23.8M 0s 2024-09-17T23:32:53.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161650K .......... .......... .......... .......... .......... 95% 51.5M 0s 2024-09-17T23:32:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161700K .......... .......... .......... .......... .......... 95% 31.1M 0s 2024-09-17T23:32:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161750K .......... .......... .......... .......... .......... 95% 26.9M 0s 2024-09-17T23:32:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161800K .......... .......... .......... .......... .......... 96% 23.7M 0s 2024-09-17T23:32:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161850K .......... .......... .......... .......... .......... 96% 57.2M 0s 2024-09-17T23:32:53.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161900K .......... .......... .......... .......... .......... 96% 23.4M 0s 2024-09-17T23:32:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:161950K .......... .......... .......... .......... .......... 96% 24.6M 0s 2024-09-17T23:32:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162000K .......... .......... .......... .......... .......... 96% 28.6M 0s 2024-09-17T23:32:53.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162050K .......... .......... .......... .......... .......... 96% 28.1M 0s 2024-09-17T23:32:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162100K .......... .......... .......... .......... .......... 96% 24.9M 0s 2024-09-17T23:32:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162150K .......... .......... .......... .......... .......... 96% 53.9M 0s 2024-09-17T23:32:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162200K .......... .......... .......... .......... .......... 96% 23.6M 0s 2024-09-17T23:32:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162250K .......... .......... .......... .......... .......... 96% 41.2M 0s 2024-09-17T23:32:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162300K .......... .......... .......... .......... .......... 96% 23.7M 0s 2024-09-17T23:32:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162350K .......... .......... .......... .......... .......... 96% 22.9M 0s 2024-09-17T23:32:53.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162400K .......... .......... .......... .......... .......... 96% 52.1M 0s 2024-09-17T23:32:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162450K .......... .......... .......... .......... .......... 96% 29.2M 0s 2024-09-17T23:32:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162500K .......... .......... .......... .......... .......... 96% 28.7M 0s 2024-09-17T23:32:53.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162550K .......... .......... .......... .......... .......... 96% 23.5M 0s 2024-09-17T23:32:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162600K .......... .......... .......... .......... .......... 96% 24.2M 0s 2024-09-17T23:32:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162650K .......... .......... .......... .......... .......... 96% 52.4M 0s 2024-09-17T23:32:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162700K .......... .......... .......... .......... .......... 96% 22.9M 0s 2024-09-17T23:32:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162750K .......... .......... .......... .......... .......... 96% 24.6M 0s 2024-09-17T23:32:53.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162800K .......... .......... .......... .......... .......... 96% 43.3M 0s 2024-09-17T23:32:53.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162850K .......... .......... .......... .......... .......... 96% 24.1M 0s 2024-09-17T23:32:53.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162900K .......... .......... .......... .......... .......... 96% 49.5M 0s 2024-09-17T23:32:53.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:162950K .......... .......... .......... .......... .......... 96% 25.9M 0s 2024-09-17T23:32:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163000K .......... .......... .......... .......... .......... 96% 29.8M 0s 2024-09-17T23:32:53.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163050K .......... .......... .......... .......... .......... 96% 25.0M 0s 2024-09-17T23:32:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163100K .......... .......... .......... .......... .......... 96% 47.1M 0s 2024-09-17T23:32:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163150K .......... .......... .......... .......... .......... 96% 15.5M 0s 2024-09-17T23:32:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163200K .......... .......... .......... .......... .......... 96% 87.7M 0s 2024-09-17T23:32:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163250K .......... .......... .......... .......... .......... 96% 25.6M 0s 2024-09-17T23:32:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163300K .......... .......... .......... .......... .......... 96% 28.7M 0s 2024-09-17T23:32:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163350K .......... .......... .......... .......... .......... 96% 24.2M 0s 2024-09-17T23:32:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163400K .......... .......... .......... .......... .......... 96% 55.0M 0s 2024-09-17T23:32:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163450K .......... .......... .......... .......... .......... 96% 25.5M 0s 2024-09-17T23:32:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163500K .......... .......... .......... .......... .......... 97% 34.7M 0s 2024-09-17T23:32:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163550K .......... .......... .......... .......... .......... 97% 23.8M 0s 2024-09-17T23:32:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163600K .......... .......... .......... .......... .......... 97% 23.7M 0s 2024-09-17T23:32:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163650K .......... .......... .......... .......... .......... 97% 46.7M 0s 2024-09-17T23:32:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163700K .......... .......... .......... .......... .......... 97% 24.2M 0s 2024-09-17T23:32:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163750K .......... .......... .......... .......... .......... 97% 32.0M 0s 2024-09-17T23:32:53.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163800K .......... .......... .......... .......... .......... 97% 24.2M 0s 2024-09-17T23:32:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163850K .......... .......... .......... .......... .......... 97% 25.9M 0s 2024-09-17T23:32:53.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163900K .......... .......... .......... .......... .......... 97% 55.2M 0s 2024-09-17T23:32:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:163950K .......... .......... .......... .......... .......... 97% 23.5M 0s 2024-09-17T23:32:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164000K .......... .......... .......... .......... .......... 97% 24.6M 0s 2024-09-17T23:32:53.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164050K .......... .......... .......... .......... .......... 97% 38.5M 0s 2024-09-17T23:32:53.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164100K .......... .......... .......... .......... .......... 97% 23.1M 0s 2024-09-17T23:32:53.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164150K .......... .......... .......... .......... .......... 97% 54.0M 0s 2024-09-17T23:32:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164200K .......... .......... .......... .......... .......... 97% 27.2M 0s 2024-09-17T23:32:53.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164250K .......... .......... .......... .......... .......... 97% 29.0M 0s 2024-09-17T23:32:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164300K .......... .......... .......... .......... .......... 97% 24.0M 0s 2024-09-17T23:32:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164350K .......... .......... .......... .......... .......... 97% 25.0M 0s 2024-09-17T23:32:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164400K .......... .......... .......... .......... .......... 97% 23.2M 0s 2024-09-17T23:32:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164450K .......... .......... .......... .......... .......... 97% 57.8M 0s 2024-09-17T23:32:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164500K .......... .......... .......... .......... .......... 97% 23.0M 0s 2024-09-17T23:32:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164550K .......... .......... .......... .......... .......... 97% 40.9M 0s 2024-09-17T23:32:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164600K .......... .......... .......... .......... .......... 97% 22.0M 0s 2024-09-17T23:32:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164650K .......... .......... .......... .......... .......... 97% 71.3M 0s 2024-09-17T23:32:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164700K .......... .......... .......... .......... .......... 97% 22.8M 0s 2024-09-17T23:32:53.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164750K .......... .......... .......... .......... .......... 97% 22.4M 0s 2024-09-17T23:32:53.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164800K .......... .......... .......... .......... .......... 97% 43.3M 0s 2024-09-17T23:32:53.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164850K .......... .......... .......... .......... .......... 97% 23.3M 0s 2024-09-17T23:32:53.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164900K .......... .......... .......... .......... .......... 97% 24.0M 0s 2024-09-17T23:32:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:164950K .......... .......... .......... .......... .......... 97% 53.2M 0s 2024-09-17T23:32:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165000K .......... .......... .......... .......... .......... 97% 23.2M 0s 2024-09-17T23:32:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165050K .......... .......... .......... .......... .......... 97% 36.2M 0s 2024-09-17T23:32:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165100K .......... .......... .......... .......... .......... 97% 25.2M 0s 2024-09-17T23:32:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165150K .......... .......... .......... .......... .......... 98% 24.1M 0s 2024-09-17T23:32:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165200K .......... .......... .......... .......... .......... 98% 45.4M 0s 2024-09-17T23:32:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165250K .......... .......... .......... .......... .......... 98% 24.7M 0s 2024-09-17T23:32:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165300K .......... .......... .......... .......... .......... 98% 37.1M 0s 2024-09-17T23:32:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165350K .......... .......... .......... .......... .......... 98% 24.9M 0s 2024-09-17T23:32:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165400K .......... .......... .......... .......... .......... 98% 23.2M 0s 2024-09-17T23:32:53.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165450K .......... .......... .......... .......... .......... 98% 55.1M 0s 2024-09-17T23:32:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165500K .......... .......... .......... .......... .......... 98% 39.1M 0s 2024-09-17T23:32:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165550K .......... .......... .......... .......... .......... 98% 16.7M 0s 2024-09-17T23:32:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165600K .......... .......... .......... .......... .......... 98% 39.9M 0s 2024-09-17T23:32:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165650K .......... .......... .......... .......... .......... 98% 24.1M 0s 2024-09-17T23:32:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165700K .......... .......... .......... .......... .......... 98% 51.3M 0s 2024-09-17T23:32:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165750K .......... .......... .......... .......... .......... 98% 22.9M 0s 2024-09-17T23:32:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165800K .......... .......... .......... .......... .......... 98% 41.8M 0s 2024-09-17T23:32:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165850K .......... .......... .......... .......... .......... 98% 23.8M 0s 2024-09-17T23:32:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165900K .......... .......... .......... .......... .......... 98% 49.5M 0s 2024-09-17T23:32:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:165950K .......... .......... .......... .......... .......... 98% 15.3M 0s 2024-09-17T23:32:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166000K .......... .......... .......... .......... .......... 98% 56.0M 0s 2024-09-17T23:32:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166050K .......... .......... .......... .......... .......... 98% 33.8M 0s 2024-09-17T23:32:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166100K .......... .......... .......... .......... .......... 98% 24.6M 0s 2024-09-17T23:32:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166150K .......... .......... .......... .......... .......... 98% 23.3M 0s 2024-09-17T23:32:53.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166200K .......... .......... .......... .......... .......... 98% 54.6M 0s 2024-09-17T23:32:53.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166250K .......... .......... .......... .......... .......... 98% 26.1M 0s 2024-09-17T23:32:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166300K .......... .......... .......... .......... .......... 98% 36.1M 0s 2024-09-17T23:32:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166350K .......... .......... .......... .......... .......... 98% 23.2M 0s 2024-09-17T23:32:53.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166400K .......... .......... .......... .......... .......... 98% 24.4M 0s 2024-09-17T23:32:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166450K .......... .......... .......... .......... .......... 98% 41.5M 0s 2024-09-17T23:32:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166500K .......... .......... .......... .......... .......... 98% 26.4M 0s 2024-09-17T23:32:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166550K .......... .......... .......... .......... .......... 98% 31.9M 0s 2024-09-17T23:32:53.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166600K .......... .......... .......... .......... .......... 98% 25.4M 0s 2024-09-17T23:32:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166650K .......... .......... .......... .......... .......... 98% 25.2M 0s 2024-09-17T23:32:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166700K .......... .......... .......... .......... .......... 98% 44.2M 0s 2024-09-17T23:32:53.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166750K .......... .......... .......... .......... .......... 98% 26.2M 0s 2024-09-17T23:32:53.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166800K .......... .......... .......... .......... .......... 98% 22.9M 0s 2024-09-17T23:32:53.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166850K .......... .......... .......... .......... .......... 99% 39.8M 0s 2024-09-17T23:32:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166900K .......... .......... .......... .......... .......... 99% 24.6M 0s 2024-09-17T23:32:53.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:166950K .......... .......... .......... .......... .......... 99% 51.1M 0s 2024-09-17T23:32:53.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167000K .......... .......... .......... .......... .......... 99% 23.3M 0s 2024-09-17T23:32:53.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167050K .......... .......... .......... .......... .......... 99% 41.0M 0s 2024-09-17T23:32:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167100K .......... .......... .......... .......... .......... 99% 21.5M 0s 2024-09-17T23:32:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167150K .......... .......... .......... .......... .......... 99% 26.0M 0s 2024-09-17T23:32:53.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167200K .......... .......... .......... .......... .......... 99% 19.6M 0s 2024-09-17T23:32:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167250K .......... .......... .......... .......... .......... 99% 49.3M 0s 2024-09-17T23:32:53.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167300K .......... .......... .......... .......... .......... 99% 24.2M 0s 2024-09-17T23:32:53.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167350K .......... .......... .......... .......... .......... 99% 39.3M 0s 2024-09-17T23:32:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167400K .......... .......... .......... .......... .......... 99% 24.0M 0s 2024-09-17T23:32:53.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167450K .......... .......... .......... .......... .......... 99% 49.7M 0s 2024-09-17T23:32:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167500K .......... .......... .......... .......... .......... 99% 24.1M 0s 2024-09-17T23:32:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167550K .......... .......... .......... .......... .......... 99% 23.8M 0s 2024-09-17T23:32:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167600K .......... .......... .......... .......... .......... 99% 29.3M 0s 2024-09-17T23:32:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167650K .......... .......... .......... .......... .......... 99% 27.2M 0s 2024-09-17T23:32:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167700K .......... .......... .......... .......... .......... 99% 24.1M 0s 2024-09-17T23:32:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167750K .......... .......... .......... .......... .......... 99% 51.0M 0s 2024-09-17T23:32:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167800K .......... .......... .......... .......... .......... 99% 20.5M 0s 2024-09-17T23:32:53.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167850K .......... .......... .......... .......... .......... 99% 53.5M 0s 2024-09-17T23:32:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167900K .......... .......... .......... .......... .......... 99% 24.2M 0s 2024-09-17T23:32:53.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:167950K .......... .......... .......... .......... .......... 99% 23.0M 0s 2024-09-17T23:32:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:168000K .......... .......... .......... .......... .......... 99% 52.6M 0s 2024-09-17T23:32:53.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:168050K .......... .......... .......... .......... .......... 99% 23.8M 0s 2024-09-17T23:32:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:168100K .......... .......... .......... .......... .......... 99% 40.2M 0s 2024-09-17T23:32:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:168150K .......... .......... .......... .......... .......... 99% 22.6M 0s 2024-09-17T23:32:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:168200K .......... .......... .......... .......... .......... 99% 24.0M 0s 2024-09-17T23:32:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:168250K .......... .......... .......... .......... .......... 99% 48.7M 0s 2024-09-17T23:32:53.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:168300K .......... .......... .......... .......... .......... 99% 34.5M 0s 2024-09-17T23:32:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:168350K .......... .......... .......... .......... .......... 99% 18.4M 0s 2024-09-17T23:32:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:168400K .......... .......... .......... .......... .......... 99% 413K 0s 2024-09-17T23:32:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:168450K .......... .......... .......... .......... .......... 99% 2.37M 0s 2024-09-17T23:32:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:168500K .......... .......... .......... .......... .......... 99% 441M 0s 2024-09-17T23:32:53.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:168550K .......... .......... . 100% 399M=5.9s 2024-09-17T23:32:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-09-17T23:32:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:2024-09-17 23:32:53 (28.0 MB/s) - ‘linux-5.4.tar.gz’ saved [172616875/172616875] 2024-09-17T23:32:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: 2024-09-17T23:32:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..10} 2024-09-17T23:32:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS=() 2024-09-17T23:32:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS=() 2024-09-17T23:32:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS=() 2024-09-17T23:32:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS=() 2024-09-17T23:32:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ RET=0 2024-09-17T23:32:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:32:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-17T23:32:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary1 2024-09-17T23:32:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-17T23:32:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:32:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:32:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:32:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:53.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-17T23:32:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:32:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:32:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:32:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:32:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-17T23:32:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:32:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:32:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:32:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-17T23:32:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:32:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:32:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-17T23:32:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:32:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-17T23:32:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary1': 2024-09-17T23:32:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:32:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:32:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:32:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10b69d64a50b 2024-09-17T23:32:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10b69d64a50b 2024-09-17T23:32:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:32:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:32:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:32:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:32:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:32:53 2024 2024-09-17T23:32:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:32:53 2024 2024-09-17T23:32:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:32:53 2024 2024-09-17T23:32:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:32:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:32:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: f30e9a93-a05d-4762-895f-d20592ac2dcb 2024-09-17T23:32:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:32:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:32:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:32:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-17T23:32:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd0 2024-09-17T23:32:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:32:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-17T23:32:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:32:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:32:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:32:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 61ec46d2-9ee9-46fd-afd3-d0ae730b5161 2024-09-17T23:32:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:32:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:32:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:32:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:32:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:32:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:32:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:32:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:32:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary1 2024-09-17T23:32:54.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-17T23:32:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:32:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary1 2024-09-17T23:32:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:32:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-17T23:32:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:32:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:32:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:32:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:32:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:32:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-17T23:32:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary2 2024-09-17T23:32:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:32:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary1 2024-09-17T23:32:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-17T23:32:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:32:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary1 2024-09-17T23:32:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:32:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-17T23:32:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:32:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:32:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-17T23:32:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:32:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:32:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:32:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:32:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-17T23:32:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:32:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:32:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:32:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-17T23:32:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:32:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:32:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-17T23:32:54.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:32:54.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-17T23:32:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:32:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 4 2024-09-17T23:32:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:32:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-17T23:32:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary2': 2024-09-17T23:32:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:32:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:32:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:32:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10c634f5a635 2024-09-17T23:32:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10c634f5a635 2024-09-17T23:32:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:32:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:32:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:32:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:32:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:32:54 2024 2024-09-17T23:32:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:32:54 2024 2024-09-17T23:32:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:32:54 2024 2024-09-17T23:32:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:32:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:32:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 7b92e73e-2de8-435e-8340-c980306d099c 2024-09-17T23:32:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:32:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:32:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:32:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-17T23:32:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd1 2024-09-17T23:32:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:32:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-17T23:32:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:32:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:32:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:32:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 0e372998-0761-47bc-a025-5a76e4d00490 2024-09-17T23:32:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:32:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:32:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:32:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:32:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:32:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:32:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:32:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:32:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary2 2024-09-17T23:32:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-17T23:32:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:32:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary2 2024-09-17T23:32:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:32:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-17T23:32:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:32:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:32:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:32:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-17T23:32:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary3 2024-09-17T23:32:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:32:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:32:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-17T23:32:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary2 2024-09-17T23:32:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:32:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:32:55.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:32:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:32:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:32:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary2 2024-09-17T23:32:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-17T23:32:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-17T23:32:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:32:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:32:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:32:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:32:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-17T23:32:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:32:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-17T23:32:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:32:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:32:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:32:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-17T23:32:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:32:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:32:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-17T23:32:56.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:32:56.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 6 2024-09-17T23:32:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-17T23:32:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:32:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary3': 2024-09-17T23:32:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:32:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:32:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:32:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10d553786365 2024-09-17T23:32:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10d553786365 2024-09-17T23:32:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:32:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:32:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:32:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:32:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:32:55 2024 2024-09-17T23:32:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:32:55 2024 2024-09-17T23:32:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:32:55 2024 2024-09-17T23:32:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:32:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:32:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: e77d98ed-c621-4c06-8624-da95d40e9209 2024-09-17T23:32:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:32:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:32:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:32:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-17T23:32:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd2 2024-09-17T23:32:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:32:56.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-17T23:32:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:32:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:32:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:32:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: bf880814-625c-42b6-88be-5446184a82af 2024-09-17T23:32:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:32:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:32:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:32:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:32:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:32:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:32:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:32:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:32:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary3 2024-09-17T23:32:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-17T23:32:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:32:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary3 2024-09-17T23:32:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:32:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-17T23:32:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:32:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:32:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:32:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:32:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:32:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-17T23:32:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:32:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary4 2024-09-17T23:32:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:32:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-17T23:32:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary3 2024-09-17T23:32:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:32:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:32:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:32:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-17T23:32:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary3 2024-09-17T23:32:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-17T23:32:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:32:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:32:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:32:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:32:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-17T23:32:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:32:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:32:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:32:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-17T23:32:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:32:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:32:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-17T23:32:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:32:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-17T23:32:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:32:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 8 2024-09-17T23:32:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-17T23:32:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:32:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary4': 2024-09-17T23:32:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:32:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:32:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:32:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10e4664d2f4 2024-09-17T23:32:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10e4664d2f4 2024-09-17T23:32:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:32:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:32:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:32:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:32:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:32:56 2024 2024-09-17T23:32:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:32:56 2024 2024-09-17T23:32:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:32:56 2024 2024-09-17T23:32:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:32:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:32:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 4b621ebc-252c-483b-87b9-ccdacbc72bd1 2024-09-17T23:32:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:32:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:32:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:32:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-17T23:32:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd3 2024-09-17T23:32:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:32:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-17T23:32:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:32:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:32:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:32:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 2f92b2a9-f5ae-432a-bc95-c41942dd8bf9 2024-09-17T23:32:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:32:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:32:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:32:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:32:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:32:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:32:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:32:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:32:57.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary4 2024-09-17T23:32:57.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-17T23:32:57.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:32:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary4 2024-09-17T23:32:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:32:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-17T23:32:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:32:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:32:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:32:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:32:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:32:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-17T23:32:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary5 2024-09-17T23:32:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:32:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary4 2024-09-17T23:32:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-17T23:32:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:32:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:32:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:32:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary4 2024-09-17T23:32:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:32:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-17T23:32:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-17T23:32:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:32:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:32:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:32:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:32:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-17T23:32:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:32:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:32:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:32:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-17T23:32:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:32:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:32:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-17T23:32:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:32:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-17T23:32:58.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:32:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:32:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:32:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:32:58.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:32:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-17T23:32:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 10 2024-09-17T23:32:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:32:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary5': 2024-09-17T23:32:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:32:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:32:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:32:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 10f3a24cdc6b 2024-09-17T23:32:58.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.10f3a24cdc6b 2024-09-17T23:32:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:32:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:32:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:32:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:32:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:32:57 2024 2024-09-17T23:32:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:32:57 2024 2024-09-17T23:32:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:32:57 2024 2024-09-17T23:32:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:32:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:32:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 224b1b66-617c-4ba4-b6f3-1c7207f1fadc 2024-09-17T23:32:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:32:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:32:58.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:32:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-17T23:32:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd4 2024-09-17T23:32:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:32:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-17T23:32:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:32:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:32:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:32:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: bb39baca-cfc7-448a-ae5f-1538bf203076 2024-09-17T23:32:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:32:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:32:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:32:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:32:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:32:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:32:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:32:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:32:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary5 2024-09-17T23:32:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-17T23:32:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:32:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary5 2024-09-17T23:32:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:32:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-17T23:32:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:32:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:32:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:32:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:32:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:32:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-17T23:32:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:32:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary6 2024-09-17T23:32:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:32:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-17T23:32:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary5 2024-09-17T23:32:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:32:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:32:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:32:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary5 2024-09-17T23:32:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-17T23:32:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-17T23:32:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:32:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:32:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:32:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:32:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-17T23:32:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:32:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:32:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:32:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-17T23:32:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:32:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:32:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-17T23:32:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:32:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-17T23:32:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:32:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 13 2024-09-17T23:32:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:32:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:32:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:32:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:32:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-17T23:32:59.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:32:59.485 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:32:59.481+0000 7fc275b74640 -1 reset not still connected to 0x559cbd8d1110 2024-09-17T23:32:59.485 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:32:59.481+0000 7fc275b74640 -1 reset not still connected to 0x559cbda835f0 2024-09-17T23:32:59.485 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:32:59.483+0000 7fc275b74640 -1 reset not still connected to 0x559cbe51b450 2024-09-17T23:32:59.485 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:32:59.483+0000 7fc275b74640 -1 reset not still connected to 0x559cbebc1110 2024-09-17T23:32:59.485 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:32:59.483+0000 7fc275b74640 -1 reset not still connected to 0x559cbebc1520 2024-09-17T23:32:59.485 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:32:59.483+0000 7fc275b74640 -1 reset not still connected to 0x559cbebc15f0 2024-09-17T23:32:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary6': 2024-09-17T23:32:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:32:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:32:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:32:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1105612e7dcd 2024-09-17T23:32:59.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1105612e7dcd 2024-09-17T23:32:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:32:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:32:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:32:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:32:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:32:58 2024 2024-09-17T23:32:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:32:58 2024 2024-09-17T23:32:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:32:58 2024 2024-09-17T23:32:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:32:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:32:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9960a41a-881d-4c48-aa63-09f329f03773 2024-09-17T23:32:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:32:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:32:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:32:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-17T23:32:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd5 2024-09-17T23:32:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:32:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-17T23:32:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:32:59.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:32:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:32:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 9dc55688-9ad7-4498-ac99-5b9a0b873650 2024-09-17T23:32:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:32:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:32:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:32:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:32:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:32:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:32:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:32:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:32:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary6 2024-09-17T23:32:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-17T23:32:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:32:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary6 2024-09-17T23:32:59.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:32:59.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-17T23:32:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:32:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:32:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:32:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-17T23:32:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary7 2024-09-17T23:32:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:32:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary6 2024-09-17T23:32:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-17T23:32:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:32:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:59.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:32:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:32:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:32:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary6 2024-09-17T23:32:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:32:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:32:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-17T23:32:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-17T23:32:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:32:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:32:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:32:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:32:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:32:59.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-17T23:32:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:32:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:32:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:32:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-17T23:32:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:32:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:32:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:32:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:32:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-17T23:32:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:32:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-17T23:33:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 12 2024-09-17T23:33:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:33:00.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:00.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:00.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:33:00.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:33:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:33:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 16 2024-09-17T23:33:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-17T23:33:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary7': 2024-09-17T23:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1116bed2a05d 2024-09-17T23:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1116bed2a05d 2024-09-17T23:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:32:59 2024 2024-09-17T23:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:32:59 2024 2024-09-17T23:33:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:32:59 2024 2024-09-17T23:33:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:33:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:33:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 3e92b0f8-d381-4971-92c7-09af435faee0 2024-09-17T23:33:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:33:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:33:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:33:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-17T23:33:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd6 2024-09-17T23:33:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:33:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-17T23:33:00.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:33:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:33:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:33:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 6999ec07-67ef-4886-913b-b20932cb4509 2024-09-17T23:33:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:33:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:33:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:33:00.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:33:00.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:33:00.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:33:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:33:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:33:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary7 2024-09-17T23:33:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-17T23:33:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:33:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary7 2024-09-17T23:33:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:33:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-17T23:33:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:33:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:33:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:33:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-17T23:33:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:33:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary8 2024-09-17T23:33:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-17T23:33:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:33:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary7 2024-09-17T23:33:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:33:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:33:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:33:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-17T23:33:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary7 2024-09-17T23:33:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-17T23:33:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:33:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:33:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:33:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-17T23:33:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:33:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:33:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:33:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-17T23:33:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:33:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-17T23:33:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:33:00.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-17T23:33:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 15 2024-09-17T23:33:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:33:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:33:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:33:01.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:01.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:33:01.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 19 2024-09-17T23:33:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-17T23:33:01.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary8': 2024-09-17T23:33:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:33:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:33:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:33:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 11284a5ce175 2024-09-17T23:33:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.11284a5ce175 2024-09-17T23:33:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:33:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:33:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:33:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:33:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:33:00 2024 2024-09-17T23:33:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:33:00 2024 2024-09-17T23:33:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:33:00 2024 2024-09-17T23:33:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:33:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:33:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: a6f40b0d-defd-4e24-86c9-89901edafb97 2024-09-17T23:33:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:33:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:33:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:33:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-17T23:33:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd7 2024-09-17T23:33:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:33:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-17T23:33:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:33:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:33:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:33:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 465870a4-84e2-4a32-9aef-73d97ad8e23e 2024-09-17T23:33:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:33:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:33:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:33:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:33:01.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:33:01.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:33:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:33:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:33:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary8 2024-09-17T23:33:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-17T23:33:01.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:33:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary8 2024-09-17T23:33:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:33:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-17T23:33:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:01.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:33:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:33:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:33:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:33:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary8 2024-09-17T23:33:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-17T23:33:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary9 2024-09-17T23:33:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-17T23:33:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary8 2024-09-17T23:33:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-17T23:33:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:33:01.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-17T23:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:33:01.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:01.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-17T23:33:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:33:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:33:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:33:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-17T23:33:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:33:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-17T23:33:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:33:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-17T23:33:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 18 2024-09-17T23:33:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:33:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:33:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:33:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:33:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 22 2024-09-17T23:33:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:02.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-17T23:33:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary9': 2024-09-17T23:33:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:33:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:33:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:33:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 113a568ee30f 2024-09-17T23:33:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.113a568ee30f 2024-09-17T23:33:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:33:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:33:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:33:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:33:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:33:01 2024 2024-09-17T23:33:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:33:01 2024 2024-09-17T23:33:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:33:01 2024 2024-09-17T23:33:02.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:33:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:33:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 913c3aca-74b9-4a86-9309-65b773ebf0ba 2024-09-17T23:33:02.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:33:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:33:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:33:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-17T23:33:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd8 2024-09-17T23:33:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:33:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-17T23:33:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:33:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:33:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:33:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 5c757852-f9bd-470e-a402-07b1c79849a4 2024-09-17T23:33:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:33:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:33:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:33:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:33:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:33:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:33:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:33:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:33:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary9 2024-09-17T23:33:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-17T23:33:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:33:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary9 2024-09-17T23:33:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:33:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-17T23:33:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:33:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:33:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-17T23:33:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary10 2024-09-17T23:33:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary9 2024-09-17T23:33:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:33:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-17T23:33:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary9 2024-09-17T23:33:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:33:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-17T23:33:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:33:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:33:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-17T23:33:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:33:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:33:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:33:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:33:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-17T23:33:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:33:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:33:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:33:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-17T23:33:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:33:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-17T23:33:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:33:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-17T23:33:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 21 2024-09-17T23:33:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:33:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:03.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:33:03.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:33:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:33:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:33:03.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:33:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:33:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 25 2024-09-17T23:33:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-17T23:33:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary10': 2024-09-17T23:33:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:33:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:33:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:33:03.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 114c5314b9e4 2024-09-17T23:33:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.114c5314b9e4 2024-09-17T23:33:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:33:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:33:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:33:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:33:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:33:02 2024 2024-09-17T23:33:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:33:02 2024 2024-09-17T23:33:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:33:02 2024 2024-09-17T23:33:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:33:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:33:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 4e49171c-1844-47b3-b661-4003a37d9b73 2024-09-17T23:33:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:33:03.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:33:03.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:33:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-17T23:33:03.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd9 2024-09-17T23:33:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:33:03.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-17T23:33:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:33:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:33:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:33:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 5f938aec-bafe-4af8-9edc-644581aba75e 2024-09-17T23:33:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:33:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:33:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:33:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:33:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:33:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:33:03.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:33:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:33:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary10 2024-09-17T23:33:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-17T23:33:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:33:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary10 2024-09-17T23:33:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:33:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-17T23:33:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:33:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:33:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 35911 2024-09-17T23:33:03.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary10 2024-09-17T23:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary10 2024-09-17T23:33:03.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-17T23:33:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:33:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-17T23:33:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 24 2024-09-17T23:33:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 29 2024-09-17T23:33:04.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:33:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:33:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:33:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:33:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:33:04.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:33:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 27 2024-09-17T23:33:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 28 2024-09-17T23:33:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:33:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:33:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:33:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:33:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:33:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 31 2024-09-17T23:33:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:33:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:33:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:33:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:33:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:33:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 30 2024-09-17T23:33:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 33 2024-09-17T23:33:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 32 2024-09-17T23:33:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:33:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:33:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:33:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:07.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:33:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:33:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 34 2024-09-17T23:33:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 35 2024-09-17T23:33:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:33:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:33:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:33:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:33:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:33:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:33:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 36 2024-09-17T23:33:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 37 2024-09-17T23:33:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:33:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:33:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:33:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 38 2024-09-17T23:33:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 39 2024-09-17T23:33:10.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:10.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:33:10.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:10.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:10.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:33:10.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:33:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:33:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:33:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:33:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:33:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:33:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:33:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:33:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:33:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 40 2024-09-17T23:33:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 41 2024-09-17T23:33:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:33:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:33:11.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:11.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:33:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:33:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:33:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:33:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:33:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 43 2024-09-17T23:33:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 42 2024-09-17T23:33:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:33:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:33:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:33:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:33:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:33:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:33:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 44 2024-09-17T23:33:13.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 45 2024-09-17T23:33:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:14.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 48 2024-09-17T23:33:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 46 2024-09-17T23:33:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 49 2024-09-17T23:33:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 47 2024-09-17T23:33:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:14.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:33:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:14.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:14.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:33:14.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:33:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:33:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:33:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:33:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:33:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:33:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:33:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:33:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:33:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 50 2024-09-17T23:33:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:33:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:33:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:33:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:33:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:33:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:33:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:33:17.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:33:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:17.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:33:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:33:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:33:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:33:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:33:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:33:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:33:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:17.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:33:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 53 2024-09-17T23:33:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 52 2024-09-17T23:33:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 51 2024-09-17T23:33:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 56 2024-09-17T23:33:18.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 55 2024-09-17T23:33:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 57 2024-09-17T23:33:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 58 2024-09-17T23:33:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 59 2024-09-17T23:33:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 54 2024-09-17T23:33:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:33:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:33:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:33:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:33:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:33:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:33:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:33:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:33:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:33:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:33:20.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:33:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 62 2024-09-17T23:33:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:33:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:33:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:33:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:33:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:33:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:33:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:33:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:33:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:33:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:33:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:33:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:33:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:33:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:33:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:33:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:33:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 60 2024-09-17T23:33:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 61 2024-09-17T23:33:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 63 2024-09-17T23:33:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:22.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 65 2024-09-17T23:33:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 68 2024-09-17T23:33:23.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 69 2024-09-17T23:33:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 66 2024-09-17T23:33:23.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 64 2024-09-17T23:33:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 67 2024-09-17T23:33:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:33:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:33:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:33:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:33:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:33:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:33:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:33:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:33:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:33:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:33:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:33:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 71 2024-09-17T23:33:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:33:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:33:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:33:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:33:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:33:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:33:26.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:26.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:33:26.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:26.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:26.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:33:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:33:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:33:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:33:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:33:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:33:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:33:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:33:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:33:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 70 2024-09-17T23:33:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 72 2024-09-17T23:33:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 74 2024-09-17T23:33:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 75 2024-09-17T23:33:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 76 2024-09-17T23:33:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 77 2024-09-17T23:33:28.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 73 2024-09-17T23:33:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 79 2024-09-17T23:33:28.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 78 2024-09-17T23:33:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:33:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:33:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:33:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:33:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:33:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:33:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:33:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:33:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:33:31.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 80 2024-09-17T23:33:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 82 2024-09-17T23:33:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:31.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:33:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:33:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:33:31.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:31.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:33:31.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:31.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:33:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:33:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:33:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:33:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:33:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:33:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:33:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:33:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:33:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:33:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:33:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:33:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:33:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:33:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 81 2024-09-17T23:33:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 86 2024-09-17T23:33:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 89 2024-09-17T23:33:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 84 2024-09-17T23:33:33.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 83 2024-09-17T23:33:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 87 2024-09-17T23:33:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 85 2024-09-17T23:33:33.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 88 2024-09-17T23:33:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:33:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:33:34.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:33:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:33:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:33:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:33:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:33:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:33:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:33:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:33:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:33:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 90 2024-09-17T23:33:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 91 2024-09-17T23:33:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:33:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:33:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:33:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:33:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:33:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:33:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:33:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:33:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:33:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:33:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:33:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:33:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:33:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:33:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:33:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:33:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 93 2024-09-17T23:33:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 92 2024-09-17T23:33:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 99 2024-09-17T23:33:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:38.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 95 2024-09-17T23:33:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 98 2024-09-17T23:33:38.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 94 2024-09-17T23:33:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 97 2024-09-17T23:33:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 96 2024-09-17T23:33:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:33:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:33:39.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:33:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:33:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:33:39.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:33:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:33:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:33:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:33:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:33:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:33:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:33:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:33:41.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 100 2024-09-17T23:33:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 101 2024-09-17T23:33:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:33:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:33:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:33:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:33:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:33:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:33:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:33:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:33:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:33:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:33:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:33:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:33:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:33:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:33:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 103 2024-09-17T23:33:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 102 2024-09-17T23:33:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 104 2024-09-17T23:33:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 109 2024-09-17T23:33:42.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 105 2024-09-17T23:33:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 107 2024-09-17T23:33:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:43.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 108 2024-09-17T23:33:44.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 106 2024-09-17T23:33:44.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:33:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:33:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:33:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:33:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:33:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:33:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:33:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:33:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:33:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:33:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:33:45.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:33:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:45.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:33:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:33:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:33:45.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:33:46.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 110 2024-09-17T23:33:46.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 111 2024-09-17T23:33:46.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:33:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:33:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:33:46.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:33:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:46.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:33:46.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:33:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:33:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:33:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:33:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:33:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:33:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 112 2024-09-17T23:33:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 115 2024-09-17T23:33:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 113 2024-09-17T23:33:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 119 2024-09-17T23:33:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 118 2024-09-17T23:33:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:47.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 114 2024-09-17T23:33:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 117 2024-09-17T23:33:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 116 2024-09-17T23:33:48.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:33:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:33:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:33:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:33:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:33:49.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:33:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:33:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:33:50.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:33:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:33:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:33:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:33:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:33:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:33:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:33:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:50.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:33:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:33:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:33:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:33:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 120 2024-09-17T23:33:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 121 2024-09-17T23:33:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:33:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:33:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:33:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:33:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:33:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:33:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:33:51.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:33:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 126 2024-09-17T23:33:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 123 2024-09-17T23:33:52.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 122 2024-09-17T23:33:52.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 125 2024-09-17T23:33:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 124 2024-09-17T23:33:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 127 2024-09-17T23:33:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 129 2024-09-17T23:33:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 128 2024-09-17T23:33:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:33:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:33:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:33:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:33:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:33:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:33:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:33:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:33:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:33:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:33:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:33:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:33:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:33:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:33:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:33:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:33:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:33:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:33:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:33:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:33:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:33:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:33:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:33:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:33:55.692 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:33:55.690+0000 7fc275b74640 -1 reset not still connected to 0x559cbe51b450 2024-09-17T23:33:55.692 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:33:55.690+0000 7fc275b74640 -1 reset not still connected to 0x559cbebc1110 2024-09-17T23:33:55.692 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:33:55.690+0000 7fc275b74640 -1 reset not still connected to 0x559cbebc15f0 2024-09-17T23:33:55.692 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:33:55.690+0000 7fc275b74640 -1 reset not still connected to 0x559cbebc16c0 2024-09-17T23:33:55.693 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:33:55.690+0000 7fc275b74640 -1 reset not still connected to 0x559cbee485b0 2024-09-17T23:33:55.693 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:33:55.690+0000 7fc275b74640 -1 reset not still connected to 0x559cbf0e1a00 2024-09-17T23:33:55.693 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:33:55.690+0000 7fc275b74640 -1 reset not still connected to 0x559cbf426680 2024-09-17T23:33:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 130 2024-09-17T23:33:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 131 2024-09-17T23:33:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:33:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:33:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:33:56.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:56.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:33:56.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:56.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:56.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:33:56.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:33:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 137 2024-09-17T23:33:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 135 2024-09-17T23:33:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 136 2024-09-17T23:33:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 134 2024-09-17T23:33:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 132 2024-09-17T23:33:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 133 2024-09-17T23:33:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:57.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:58.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 138 2024-09-17T23:33:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 139 2024-09-17T23:33:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:33:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:33:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:33:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:33:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:33:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:33:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:33:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:33:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:33:59.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:34:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:34:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:34:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:34:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:34:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:34:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:34:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:34:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:34:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:34:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:34:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:34:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:34:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:34:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:34:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:34:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:34:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:34:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:34:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 140 2024-09-17T23:34:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 141 2024-09-17T23:34:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:34:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:34:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:34:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:34:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:34:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:34:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:01.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:02.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 142 2024-09-17T23:34:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 143 2024-09-17T23:34:02.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 147 2024-09-17T23:34:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 145 2024-09-17T23:34:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:02.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:02.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 148 2024-09-17T23:34:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 144 2024-09-17T23:34:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 149 2024-09-17T23:34:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 146 2024-09-17T23:34:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:34:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:34:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:34:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:34:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:34:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:34:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:34:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:34:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:34:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:34:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:34:05.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:34:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:34:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:34:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:34:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:34:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:34:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:34:06.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 151 2024-09-17T23:34:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 150 2024-09-17T23:34:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:34:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:34:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:34:06.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:06.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:06.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:34:06.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:06.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:06.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:34:06.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:34:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:06.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:34:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:34:06.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:34:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:06.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:34:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:34:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:34:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 153 2024-09-17T23:34:07.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 152 2024-09-17T23:34:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 155 2024-09-17T23:34:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 154 2024-09-17T23:34:07.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 159 2024-09-17T23:34:07.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 158 2024-09-17T23:34:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:08.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 156 2024-09-17T23:34:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 157 2024-09-17T23:34:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:34:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:34:09.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:34:09.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:09.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:34:09.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:34:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:34:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:34:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:34:10.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:34:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:34:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:34:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:34:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:34:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:34:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:34:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:34:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:34:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:34:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:34:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:34:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:34:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:34:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:34:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:34:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 161 2024-09-17T23:34:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 160 2024-09-17T23:34:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:34:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:34:11.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:34:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:34:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:34:11.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:34:11.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 166 2024-09-17T23:34:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 167 2024-09-17T23:34:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 164 2024-09-17T23:34:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 163 2024-09-17T23:34:12.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 162 2024-09-17T23:34:12.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 169 2024-09-17T23:34:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 165 2024-09-17T23:34:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 168 2024-09-17T23:34:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:34:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:34:14.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:34:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:34:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:34:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:34:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:34:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:34:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:34:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:34:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:34:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:34:15.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:15.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:34:15.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:15.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:15.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:34:15.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:34:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:34:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:34:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:34:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:34:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:34:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:34:16.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 171 2024-09-17T23:34:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:34:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:34:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:34:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 170 2024-09-17T23:34:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:34:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:34:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:34:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:34:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:34:16.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:34:16.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 174 2024-09-17T23:34:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 176 2024-09-17T23:34:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 172 2024-09-17T23:34:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 175 2024-09-17T23:34:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 173 2024-09-17T23:34:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 178 2024-09-17T23:34:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 177 2024-09-17T23:34:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 179 2024-09-17T23:34:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:34:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:34:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:34:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:34:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:34:19.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:34:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:34:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:34:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:34:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:34:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:34:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:34:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:34:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:34:20.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:34:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:34:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:34:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:34:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:34:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:34:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:34:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:34:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:34:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:34:21.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 180 2024-09-17T23:34:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 181 2024-09-17T23:34:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:34:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:34:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:34:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:34:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:34:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:34:21.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 185 2024-09-17T23:34:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 182 2024-09-17T23:34:22.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 183 2024-09-17T23:34:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 187 2024-09-17T23:34:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 186 2024-09-17T23:34:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 184 2024-09-17T23:34:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 189 2024-09-17T23:34:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 188 2024-09-17T23:34:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:34:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:34:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:34:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:34:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:34:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:34:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:34:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:34:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:34:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:34:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:34:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:34:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:34:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:34:25.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:34:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:34:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:34:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:34:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:34:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:34:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:34:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:34:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:34:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:34:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 191 2024-09-17T23:34:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 190 2024-09-17T23:34:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:34:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:34:26.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:34:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:34:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:34:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:34:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 195 2024-09-17T23:34:27.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 197 2024-09-17T23:34:27.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 194 2024-09-17T23:34:27.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 192 2024-09-17T23:34:27.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 196 2024-09-17T23:34:27.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 193 2024-09-17T23:34:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 198 2024-09-17T23:34:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 199 2024-09-17T23:34:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:28.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:34:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:34:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:34:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:34:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:34:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:34:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:34:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:34:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:34:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:34:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:34:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:34:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:34:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:34:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:34:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:34:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:34:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:34:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:34:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:34:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:34:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:34:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:34:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:34:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 200 2024-09-17T23:34:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 201 2024-09-17T23:34:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:34:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:34:31.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:34:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:34:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:34:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:34:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 202 2024-09-17T23:34:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 203 2024-09-17T23:34:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 205 2024-09-17T23:34:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 206 2024-09-17T23:34:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 204 2024-09-17T23:34:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 207 2024-09-17T23:34:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:32.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 208 2024-09-17T23:34:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 209 2024-09-17T23:34:33.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:34:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:34:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:34:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:34:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:34:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:34:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:34:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:34:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:34:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:34:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:34:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:34:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:34:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:34:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:34:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:34:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:34:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:34:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:34:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:34:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:34:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:34:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:34:35.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:34:36.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 211 2024-09-17T23:34:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 210 2024-09-17T23:34:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:34:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:34:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:34:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:34:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:34:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:34:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 214 2024-09-17T23:34:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 215 2024-09-17T23:34:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 212 2024-09-17T23:34:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 216 2024-09-17T23:34:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 217 2024-09-17T23:34:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:38.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 218 2024-09-17T23:34:38.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 219 2024-09-17T23:34:38.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 213 2024-09-17T23:34:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:34:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:34:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:34:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:34:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:34:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:34:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:34:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:34:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:34:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:34:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:40.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:34:40.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:34:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:34:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:34:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:34:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:34:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:34:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:34:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:34:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:34:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:34:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 221 2024-09-17T23:34:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 220 2024-09-17T23:34:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:34:41.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:34:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:34:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:34:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:34:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:34:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:34:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:34:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:34:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 222 2024-09-17T23:34:42.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 225 2024-09-17T23:34:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 223 2024-09-17T23:34:42.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 224 2024-09-17T23:34:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 226 2024-09-17T23:34:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:42.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 229 2024-09-17T23:34:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:43.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 227 2024-09-17T23:34:43.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 228 2024-09-17T23:34:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:34:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:44.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:34:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:34:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:34:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:34:44.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:34:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:34:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:34:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:34:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:34:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:34:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:34:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:34:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:34:45.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:34:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:34:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:34:45.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:34:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:34:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:34:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:34:45.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:34:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:34:45.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:34:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 231 2024-09-17T23:34:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 230 2024-09-17T23:34:46.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:46.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:34:46.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:46.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:46.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:34:46.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:34:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:34:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:34:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:34:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:47.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 234 2024-09-17T23:34:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 233 2024-09-17T23:34:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 232 2024-09-17T23:34:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 237 2024-09-17T23:34:47.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 236 2024-09-17T23:34:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 235 2024-09-17T23:34:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:47.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 238 2024-09-17T23:34:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 239 2024-09-17T23:34:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:34:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:34:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:34:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:49.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:34:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:34:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:34:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:34:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:34:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:34:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:34:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:34:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:34:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:34:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:34:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:34:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:34:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:34:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:34:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:34:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:34:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:34:50.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:34:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 240 2024-09-17T23:34:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:34:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:34:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:34:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 241 2024-09-17T23:34:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:34:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:34:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:34:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:51.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 243 2024-09-17T23:34:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 244 2024-09-17T23:34:52.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 245 2024-09-17T23:34:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 246 2024-09-17T23:34:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 247 2024-09-17T23:34:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 242 2024-09-17T23:34:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:52.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 248 2024-09-17T23:34:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 250 2024-09-17T23:34:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:34:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:34:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:34:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:34:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:34:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:34:55.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:34:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:34:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:34:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:34:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:34:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:34:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:34:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:34:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:34:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:34:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:34:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:34:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:34:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:34:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:34:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:34:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:34:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:34:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 252 2024-09-17T23:34:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 251 2024-09-17T23:34:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:34:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:34:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:34:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:34:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:34:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:34:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 258 2024-09-17T23:34:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 253 2024-09-17T23:34:57.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 255 2024-09-17T23:34:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 256 2024-09-17T23:34:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 254 2024-09-17T23:34:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 257 2024-09-17T23:34:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:58.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 260 2024-09-17T23:34:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 259 2024-09-17T23:34:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:58.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:34:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:34:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:34:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:34:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:34:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:34:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:34:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:34:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:34:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:35:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:35:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:35:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:35:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:35:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:35:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:35:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:35:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:35:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:35:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:35:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:35:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:35:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:35:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:35:00.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:35:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:35:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:35:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:35:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 261 2024-09-17T23:35:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:35:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:35:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:35:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:35:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:35:01.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:35:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 262 2024-09-17T23:35:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:01.786 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:35:01.783+0000 7f9bfab49640 -1 reset not still connected to 0x55cd08730000 2024-09-17T23:35:01.786 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:35:01.783+0000 7f9bfab49640 -1 reset not still connected to 0x55cd093336c0 2024-09-17T23:35:01.816 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:35:01.814+0000 7f9bfab49640 -1 reset not still connected to 0x55cd07dce1a0 2024-09-17T23:35:01.816 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:35:01.814+0000 7f9bfab49640 -1 reset not still connected to 0x55cd07f3b520 2024-09-17T23:35:01.817 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:35:01.815+0000 7f9bfab49640 -1 reset not still connected to 0x55cd09333d40 2024-09-17T23:35:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 266 2024-09-17T23:35:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 265 2024-09-17T23:35:02.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 268 2024-09-17T23:35:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 263 2024-09-17T23:35:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 264 2024-09-17T23:35:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:02.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:03.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 269 2024-09-17T23:35:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 270 2024-09-17T23:35:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 267 2024-09-17T23:35:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:03.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:35:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:35:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:35:04.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:04.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:35:04.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:04.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:04.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:35:04.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:35:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:35:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:35:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:35:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:35:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:35:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:35:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:35:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:35:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:35:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:35:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:35:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:35:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:35:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:35:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:35:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 272 2024-09-17T23:35:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 271 2024-09-17T23:35:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:35:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:35:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:35:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:35:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:35:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:35:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:35:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:35:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:35:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 274 2024-09-17T23:35:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 276 2024-09-17T23:35:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 273 2024-09-17T23:35:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 277 2024-09-17T23:35:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 275 2024-09-17T23:35:07.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:07.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 279 2024-09-17T23:35:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:08.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 280 2024-09-17T23:35:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 278 2024-09-17T23:35:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:35:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:35:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:35:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:35:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:35:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:35:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:35:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:35:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:35:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:35:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:35:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:35:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:35:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:35:10.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:35:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:35:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:35:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:35:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:35:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:35:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:35:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:35:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:35:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:35:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 281 2024-09-17T23:35:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:35:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:35:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:35:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:35:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:35:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:35:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 282 2024-09-17T23:35:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 286 2024-09-17T23:35:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 287 2024-09-17T23:35:12.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 285 2024-09-17T23:35:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 284 2024-09-17T23:35:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 288 2024-09-17T23:35:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 283 2024-09-17T23:35:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 289 2024-09-17T23:35:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 290 2024-09-17T23:35:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:35:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:14.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:35:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:35:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:35:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:35:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:35:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:35:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:35:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:35:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:35:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:35:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:35:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:35:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:35:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:35:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:35:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:15.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:35:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:35:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:35:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:35:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:35:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:35:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:35:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:35:16.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 291 2024-09-17T23:35:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 292 2024-09-17T23:35:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:35:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:35:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:35:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:35:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:35:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:35:16.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:16.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:17.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 296 2024-09-17T23:35:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 297 2024-09-17T23:35:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 298 2024-09-17T23:35:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 295 2024-09-17T23:35:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 293 2024-09-17T23:35:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 294 2024-09-17T23:35:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:17.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 299 2024-09-17T23:35:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 300 2024-09-17T23:35:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:35:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:35:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:35:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:35:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:35:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:35:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:35:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:35:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:35:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:35:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 36043 2024-09-17T23:35:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:35:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:35:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:35:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:35:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:35:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:35:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:35:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:35:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:35:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:35:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:35:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:35:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:35:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:35:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:35:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 302 2024-09-17T23:35:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 301 2024-09-17T23:35:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:35:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:35:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:35:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 306 2024-09-17T23:35:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 305 2024-09-17T23:35:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 303 2024-09-17T23:35:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 307 2024-09-17T23:35:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 304 2024-09-17T23:35:22.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:22.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:22.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:23.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 308 2024-09-17T23:35:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 309 2024-09-17T23:35:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:35:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:35:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:35:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:35:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 36183 2024-09-17T23:35:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:35:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 36315 2024-09-17T23:35:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:35:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:35:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:35:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:35:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:35:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:35:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:35:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:35:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:35:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:35:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:35:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:35:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 310 2024-09-17T23:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:35:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 36467 2024-09-17T23:35:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:35:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 36619 2024-09-17T23:35:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:35:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 36772 2024-09-17T23:35:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 311 2024-09-17T23:35:27.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 312 2024-09-17T23:35:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 313 2024-09-17T23:35:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:27.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:35:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:35:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 36925 2024-09-17T23:35:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:35:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 37078 2024-09-17T23:35:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:35:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 37251 2024-09-17T23:35:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-17T23:35:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:35:30.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 35912 2024-09-17T23:37:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:37:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:37:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:37:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 36044 2024-09-17T23:37:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:37:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:37:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:37:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 36184 2024-09-17T23:37:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:37:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:37:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:37:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 36316 2024-09-17T23:37:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:37:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:37:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:37:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 36468 2024-09-17T23:37:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:37:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:37:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:37:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 36620 2024-09-17T23:37:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:37:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:37:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:37:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 36773 2024-09-17T23:38:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:38:00.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:38:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:38:00.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 36926 2024-09-17T23:38:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:38:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:38:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:38:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 37079 2024-09-17T23:38:02.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:38:02.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:38:02.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:38:02.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 37252 2024-09-17T23:38:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:38:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:38:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-17T23:38:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:38:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:38:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:38:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 1 2024-09-17T23:38:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd0 2024-09-17T23:38:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary1 2024-09-17T23:38:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary1 2024-09-17T23:38:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:38:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:38:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary1 2024-09-17T23:38:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:38:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 2 2024-09-17T23:38:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd1 2024-09-17T23:38:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary2 2024-09-17T23:38:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary2 2024-09-17T23:38:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:38:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary2 2024-09-17T23:38:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:38:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:38:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 3 2024-09-17T23:38:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd2 2024-09-17T23:38:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary3 2024-09-17T23:38:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary3 2024-09-17T23:38:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:38:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary3 2024-09-17T23:38:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:38:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:38:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 4 2024-09-17T23:38:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd3 2024-09-17T23:38:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary4 2024-09-17T23:38:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary4 2024-09-17T23:38:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:38:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:38:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary4 2024-09-17T23:38:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:38:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 5 2024-09-17T23:38:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd4 2024-09-17T23:38:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:38:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary5 2024-09-17T23:38:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary5 2024-09-17T23:38:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:38:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:38:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary5 2024-09-17T23:38:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 6 2024-09-17T23:38:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd5 2024-09-17T23:38:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:38:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary6 2024-09-17T23:38:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary6 2024-09-17T23:38:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:38:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary6 2024-09-17T23:38:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 7 2024-09-17T23:38:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:38:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd6 2024-09-17T23:38:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:38:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:38:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary7 2024-09-17T23:38:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary7 2024-09-17T23:38:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:38:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary7 2024-09-17T23:38:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:38:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:38:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:38:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:38:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 44235 2024-09-17T23:38:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 10 2024-09-17T23:38:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd9 2024-09-17T23:38:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary10 2024-09-17T23:38:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary10 2024-09-17T23:38:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:38:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary10 2024-09-17T23:38:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 9 2024-09-17T23:38:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd8 2024-09-17T23:38:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary9 2024-09-17T23:38:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary9 2024-09-17T23:38:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:38:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary9 2024-09-17T23:38:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 8 2024-09-17T23:38:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd7 2024-09-17T23:38:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary8 2024-09-17T23:38:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary8 2024-09-17T23:38:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:38:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary8 2024-09-17T23:38:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-17T23:38:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:04.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-17T23:38:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-17T23:38:04.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-17T23:38:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-17T23:38:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-17T23:38:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-17T23:38:04.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-17T23:38:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-17T23:38:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:04.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-17T23:38:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=b0e848de2e2067a82e01a92e931beb6c 2024-09-17T23:38:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-17T23:38:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=67014364d39bad64dcdda8336332a559 2024-09-17T23:38:34.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-17T23:38:34.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=87bd577da9a3836dc08901ba7ca925d6 2024-09-17T23:38:34.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-17T23:38:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-17T23:38:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-17T23:38:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:38:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-17T23:38:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:38:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-17T23:38:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-17T23:38:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:38:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-17T23:38:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=17787467d690d2a6db747d31b236556f 2024-09-17T23:38:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-17T23:38:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-17T23:38:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:38:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-17T23:38:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=8c0f3f87b3e3a84855298a70649b6707 2024-09-17T23:38:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-17T23:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-17T23:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:38:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-17T23:38:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=b5721ae1244d4b32dc46ddf9e4cefe44 2024-09-17T23:38:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-17T23:38:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=11eaf45cb73cf087d93c9b162e31a867 2024-09-17T23:38:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-17T23:38:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-17T23:38:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:38:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-17T23:38:35.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=86f7d7bdc278729926738d6bfd0aedee 2024-09-17T23:38:35.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-17T23:38:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-17T23:38:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:38:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-17T23:38:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-17T23:38:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:38:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-17T23:38:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=d791c9d536556294e678bf24ff7e9384 2024-09-17T23:38:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-17T23:38:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-17T23:38:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:38:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-17T23:38:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=0442f5b9f1bf63878f49d86a5e49d414 2024-09-17T23:38:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-17T23:38:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-17T23:38:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:38:35.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-17T23:38:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:38:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:38:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:38:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-17T23:38:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:38:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:38:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:38:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-17T23:38:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:38:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-17T23:38:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:38:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-17T23:38:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:38:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-17T23:38:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:38:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:38:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-17T23:38:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:38:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:38:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:38:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-17T23:38:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:38:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-17T23:38:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:38:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-17T23:38:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-17T23:38:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-17T23:38:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:38:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-17T23:38:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:38:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:38:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-17T23:38:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:38:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-17T23:38:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-17T23:38:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:38:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:38:37.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-17T23:38:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:38:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-17T23:38:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-17T23:38:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:38:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-17T23:38:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-17T23:38:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:38:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-17T23:38:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:38:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-17T23:38:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-17T23:38:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 224b1b66-617c-4ba4-b6f3-1c7207f1fadc 2024-09-17T23:38:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37 2024-09-17T23:38:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:38:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37' 2024-09-17T23:38:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:37.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:38:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:37.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:37.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-17 23:38:37' 2024-09-17T23:38:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:38:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:37' 2024-09-17T23:38:37.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-17T23:38:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9960a41a-881d-4c48-aa63-09f329f03773 2024-09-17T23:38:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37 2024-09-17T23:38:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:38:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37' 2024-09-17T23:38:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:38:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:37.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-17T23:38:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: f30e9a93-a05d-4762-895f-d20592ac2dcb 2024-09-17T23:38:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37 2024-09-17T23:38:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:38:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37' 2024-09-17T23:38:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:38:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:37.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-17T23:38:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 4b621ebc-252c-483b-87b9-ccdacbc72bd1 2024-09-17T23:38:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37 2024-09-17T23:38:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:38:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:38:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":921600.0,"last_snapshot_bytes":184320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616122,"remote_snapshot_timestamp":1726616122,"replay_state":"idle"} 2024-09-17T23:38:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:17' 2024-09-17T23:38:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-17 23:38:37' 2024-09-17T23:38:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:38:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:37' 2024-09-17T23:38:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-17T23:38:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: e77d98ed-c621-4c06-8624-da95d40e9209 2024-09-17T23:38:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37 2024-09-17T23:38:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:38:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37' 2024-09-17T23:38:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:38:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:38:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:37.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:37.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:37.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-17 23:38:37' 2024-09-17T23:38:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:38:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:37' 2024-09-17T23:38:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:37.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:37.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:37.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:37.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-17 23:38:37' 2024-09-17T23:38:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:38:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:37' 2024-09-17T23:38:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-17 23:38:37' 2024-09-17T23:38:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:38:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:37' 2024-09-17T23:38:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-17T23:38:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a6f40b0d-defd-4e24-86c9-89901edafb97 2024-09-17T23:38:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37 2024-09-17T23:38:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:38:37.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37' 2024-09-17T23:38:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:37.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:38:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:37.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:37.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-17 23:38:37' 2024-09-17T23:38:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:38:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:37' 2024-09-17T23:38:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2576384.0,"last_snapshot_bytes":5009408,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616126,"remote_snapshot_timestamp":1726616126,"replay_state":"idle"}2024-09-17 23:38:17image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3602432.0,"last_snapshot_bytes":7065600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616120,"remote_snapshot_timestamp":1726616120,"replay_state":"idle"}2024-09-17 23:38:17image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":921600.0,"last_snapshot_bytes":184320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616122,"remote_snapshot_timestamp":1726616122,"replay_state":"idle"}2024-09-17 23:38:17image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616125,"remote_snapshot_timestamp":1726616125,"replay_state":"idle"}2024-09-17 23:38:17image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1724416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616126,"remote_snapshot_timestamp":1726616126,"replay_state":"idle"}2024-09-17 23:38:17' 2024-09-17T23:38:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-17T23:38:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-17T23:38:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2576384.0,"last_snapshot_bytes":5009408,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616126,"remote_snapshot_timestamp":1726616126,"replay_state":"idle"}2024-09-17 23:38:17image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3602432.0,"last_snapshot_bytes":7065600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616120,"remote_snapshot_timestamp":1726616120,"replay_state":"idle"}2024-09-17 23:38:17image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":921600.0,"last_snapshot_bytes":184320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616122,"remote_snapshot_timestamp":1726616122,"replay_state":"idle"}2024-09-17 23:38:17image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616125,"remote_snapshot_timestamp":1726616125,"replay_state":"idle"}2024-09-17 23:38:17image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1724416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616126,"remote_snapshot_timestamp":1726616126,"replay_state":"idle"}2024-09-17 23:38:17' 2024-09-17T23:38:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-17T23:38:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-17T23:38:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:38:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-17T23:38:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2576384.0,"last_snapshot_bytes":5009408,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616126,"remote_snapshot_timestamp":1726616126,"replay_state":"idle"}2024-09-17 23:38:17image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3602432.0,"last_snapshot_bytes":7065600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616120,"remote_snapshot_timestamp":1726616120,"replay_state":"idle"}2024-09-17 23:38:17image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":921600.0,"last_snapshot_bytes":184320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616122,"remote_snapshot_timestamp":1726616122,"replay_state":"idle"}2024-09-17 23:38:17image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616125,"remote_snapshot_timestamp":1726616125,"replay_state":"idle"}2024-09-17 23:38:17image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1724416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616126,"remote_snapshot_timestamp":1726616126,"replay_state":"idle"}2024-09-17 23:38:17' 2024-09-17T23:38:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-17T23:38:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-17T23:38:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-17T23:38:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:38:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-17T23:38:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:38:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2576384.0,"last_snapshot_bytes":5009408,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616126,"remote_snapshot_timestamp":1726616126,"replay_state":"idle"}2024-09-17 23:38:17image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3602432.0,"last_snapshot_bytes":7065600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616120,"remote_snapshot_timestamp":1726616120,"replay_state":"idle"}2024-09-17 23:38:17image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":921600.0,"last_snapshot_bytes":184320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616122,"remote_snapshot_timestamp":1726616122,"replay_state":"idle"}2024-09-17 23:38:17image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616125,"remote_snapshot_timestamp":1726616125,"replay_state":"idle"}2024-09-17 23:38:17image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stopping_replayremote image demoted2024-09-17 23:38:38' 2024-09-17T23:38:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-17T23:38:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2576384.0,"last_snapshot_bytes":5009408,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616126,"remote_snapshot_timestamp":1726616126,"replay_state":"idle"}2024-09-17 23:38:17image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3602432.0,"last_snapshot_bytes":7065600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616120,"remote_snapshot_timestamp":1726616120,"replay_state":"idle"}2024-09-17 23:38:17image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":921600.0,"last_snapshot_bytes":184320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616122,"remote_snapshot_timestamp":1726616122,"replay_state":"idle"}2024-09-17 23:38:17image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stopping_replayremote image demoted2024-09-17 23:38:38image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stopping_replayremote image demoted2024-09-17 23:38:38' 2024-09-17T23:38:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-17T23:38:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-17T23:38:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-17T23:38:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:38.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-17T23:38:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:38:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-17T23:38:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:38:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2576384.0,"last_snapshot_bytes":5009408,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616126,"remote_snapshot_timestamp":1726616126,"replay_state":"idle"}2024-09-17 23:38:17image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stopping_replayremote image demoted2024-09-17 23:38:38image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stopping_replayremote image demoted2024-09-17 23:38:38image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38' 2024-09-17T23:38:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-17T23:38:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-17T23:38:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-17T23:38:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:38:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-17T23:38:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:38:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-17T23:38:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-17T23:38:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:38:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-17T23:38:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-17T23:38:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 4e49171c-1844-47b3-b661-4003a37d9b73 2024-09-17T23:38:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37 2024-09-17T23:38:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:38:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:38:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:38' 2024-09-17T23:38:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-17T23:38:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:38:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-17T23:38:38.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:38:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:38.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-17 23:38:37' 2024-09-17T23:38:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:38:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:37' 2024-09-17T23:38:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-17T23:38:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:38:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-17T23:38:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-17T23:38:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 7b92e73e-2de8-435e-8340-c980306d099c 2024-09-17T23:38:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37 2024-09-17T23:38:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:38:38.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:38:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:38' 2024-09-17T23:38:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:38.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:38:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:38.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:38.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:38.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-17 23:38:37' 2024-09-17T23:38:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:38:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:37' 2024-09-17T23:38:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:38.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-17T23:38:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 3e92b0f8-d381-4971-92c7-09af435faee0 2024-09-17T23:38:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37 2024-09-17T23:38:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:38:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:38:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:38' 2024-09-17T23:38:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:38:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:38.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:38.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-17 23:38:37' 2024-09-17T23:38:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:38:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:37' 2024-09-17T23:38:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-17T23:38:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 913c3aca-74b9-4a86-9309-65b773ebf0ba 2024-09-17T23:38:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37 2024-09-17T23:38:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:38:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:38:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:38' 2024-09-17T23:38:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:38:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-17 23:38:37' 2024-09-17T23:38:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:38:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:38:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:37' 2024-09-17T23:38:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38' 2024-09-17T23:38:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-17T23:38:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-17T23:38:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-17T23:38:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:38:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38' 2024-09-17T23:38:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-17T23:38:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-17T23:38:38.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-17T23:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38' 2024-09-17T23:38:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:38.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-17T23:38:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-17T23:38:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:38:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:38:38' 2024-09-17T23:38:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-17T23:38:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-17T23:38:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:38:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-17T23:38:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-17T23:38:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:38:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:38:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:38:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-17T23:38:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:38:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-17T23:38:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-17T23:38:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:38:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-17T23:38:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-17T23:38:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:38:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-17T23:38:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-17T23:38:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:38:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-17T23:38:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-17T23:38:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:38:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-17T23:38:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-17T23:38:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 224b1b66-617c-4ba4-b6f3-1c7207f1fadc 2024-09-17T23:38:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37 2024-09-17T23:38:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:38:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37' 2024-09-17T23:38:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:38:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:39.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:39.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:39.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:39.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:39.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-17T23:38:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:38:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-17T23:38:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-17 23:38:37' 2024-09-17T23:38:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:38:39.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:37' 2024-09-17T23:38:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-17T23:38:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: f30e9a93-a05d-4762-895f-d20592ac2dcb 2024-09-17T23:38:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37 2024-09-17T23:38:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:38:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37' 2024-09-17T23:38:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:38:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-17 23:38:37' 2024-09-17T23:38:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:38:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:37' 2024-09-17T23:38:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-17T23:38:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9960a41a-881d-4c48-aa63-09f329f03773 2024-09-17T23:38:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37 2024-09-17T23:38:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:38:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37' 2024-09-17T23:38:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:38:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-17 23:38:37' 2024-09-17T23:38:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:38:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:37' 2024-09-17T23:38:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-17T23:38:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: e77d98ed-c621-4c06-8624-da95d40e9209 2024-09-17T23:38:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37 2024-09-17T23:38:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:38:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37' 2024-09-17T23:38:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:39.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:39.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:39.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-17T23:38:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 4b621ebc-252c-483b-87b9-ccdacbc72bd1 2024-09-17T23:38:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:38:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:38 2024-09-17T23:38:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:38:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37' 2024-09-17T23:38:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:38:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-17T23:38:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a6f40b0d-defd-4e24-86c9-89901edafb97 2024-09-17T23:38:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37 2024-09-17T23:38:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:38:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37' 2024-09-17T23:38:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:39.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:38:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:38:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-17 23:38:37' 2024-09-17T23:38:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:38:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:37' 2024-09-17T23:38:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:38' 2024-09-17T23:38:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-17 23:38:38' 2024-09-17T23:38:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:38:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:38' 2024-09-17T23:38:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-17 23:38:37' 2024-09-17T23:38:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:38:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:37' 2024-09-17T23:38:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37' 2024-09-17T23:38:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-17T23:38:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-17T23:38:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-17T23:38:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:38:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:38:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-17T23:38:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37' 2024-09-17T23:38:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:39.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-17T23:38:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:39.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-17T23:38:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-17T23:38:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:38:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:38:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-17T23:38:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37' 2024-09-17T23:38:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-17T23:38:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-17T23:38:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37' 2024-09-17T23:38:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-17T23:38:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-17T23:38:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:38:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:38:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-17T23:38:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-17T23:38:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37' 2024-09-17T23:38:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-17T23:38:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:38' 2024-09-17T23:38:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-17T23:38:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-17T23:38:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:38:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:38:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-17T23:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-17T23:38:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:38:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:38:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-17T23:38:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37' 2024-09-17T23:38:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:39.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-17T23:38:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:37' 2024-09-17T23:38:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-17T23:38:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-17T23:38:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:38:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:38:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-17T23:38:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-17T23:38:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:38:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-17T23:38:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-17T23:38:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:38:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-17T23:38:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-17T23:38:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 4e49171c-1844-47b3-b661-4003a37d9b73 2024-09-17T23:38:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:38:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:38 2024-09-17T23:38:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:38:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37' 2024-09-17T23:38:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:38:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:38' 2024-09-17T23:38:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-17 23:38:38' 2024-09-17T23:38:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:38:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:38' 2024-09-17T23:38:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-17T23:38:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 7b92e73e-2de8-435e-8340-c980306d099c 2024-09-17T23:38:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:38:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:38 2024-09-17T23:38:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:38:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37' 2024-09-17T23:38:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:38:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-17T23:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-17T23:38:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:38' 2024-09-17T23:38:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-17 23:38:38' 2024-09-17T23:38:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:38:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:38' 2024-09-17T23:38:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-17T23:38:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:38:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:38:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:38:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-17T23:38:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-17T23:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 3e92b0f8-d381-4971-92c7-09af435faee0 2024-09-17T23:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:38 2024-09-17T23:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37' 2024-09-17T23:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:39.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-17T23:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 913c3aca-74b9-4a86-9309-65b773ebf0ba 2024-09-17T23:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:38 2024-09-17T23:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:37' 2024-09-17T23:38:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:38:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:38:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:38:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:38:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:38' 2024-09-17T23:38:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:38:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-17 23:38:38' 2024-09-17T23:38:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:38:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:38:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:38:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:38' 2024-09-17T23:38:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:38:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:38:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:38:39.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:38:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:38' 2024-09-17T23:38:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-17 23:38:38' 2024-09-17T23:38:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:38:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:38:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:38' 2024-09-17T23:38:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:38:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:38:39.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:38:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37' 2024-09-17T23:38:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:39.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-17T23:38:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:38' 2024-09-17T23:38:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-17T23:38:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-17T23:38:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:38:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:38:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-17T23:38:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37' 2024-09-17T23:38:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-17T23:38:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:38' 2024-09-17T23:38:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-17T23:38:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-17T23:38:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:38:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:38:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-17T23:38:39.926 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:38:39.923+0000 7fc275b74640 -1 reset not still connected to 0x559cc0697d40 2024-09-17T23:38:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37' 2024-09-17T23:38:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-17T23:38:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:38' 2024-09-17T23:38:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-17T23:38:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary27b92e73e-2de8-435e-8340-c980306d099cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary69960a41a-881d-4c48-aa63-09f329f03773up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:38:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:38:38cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:38:37' 2024-09-17T23:38:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:38:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-17T23:38:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-17T23:38:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:38:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:38:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-17T23:38:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:38' 2024-09-17T23:38:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-17T23:38:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:38:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:38:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:38:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:38:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:38:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-17T23:38:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:38:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:38:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:38:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:38:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-17T23:38:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:38:40.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:38:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:38:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:38:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:38:40.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:38:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:38:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:38:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:38:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:38:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-17T23:38:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:38:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:38:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-17T23:38:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:38:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:38:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-17T23:38:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:38:40.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:38:40.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-17T23:38:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:38:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:38:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-17T23:38:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:38:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:38:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-17T23:38:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:38:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:38:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:38:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:38:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-17T23:38:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-17T23:38:40.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd0 2024-09-17T23:38:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-17T23:38:40.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd1 2024-09-17T23:38:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-17T23:38:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd3 2024-09-17T23:38:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-17T23:38:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd2 2024-09-17T23:38:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-17T23:38:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd4 2024-09-17T23:38:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-17T23:38:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd5 2024-09-17T23:38:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-17T23:38:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd6 2024-09-17T23:38:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-17T23:38:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd7 2024-09-17T23:38:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-17T23:38:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:40.765 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:38:40.763+0000 7fc275b74640 -1 reset not still connected to 0x559cbee73d40 2024-09-17T23:38:40.766 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:38:40.763+0000 7fc275b74640 -1 reset not still connected to 0x559cbf5dba00 2024-09-17T23:38:40.773 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:38:40.767+0000 7fc275b74640 -1 reset not still connected to 0x559cc0696ea0 2024-09-17T23:38:40.773 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:38:40.767+0000 7fc275b74640 -1 reset not still connected to 0x559cc0697c70 2024-09-17T23:38:40.773 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:38:40.767+0000 7fc275b74640 -1 reset not still connected to 0x559cc0ee6ea0 2024-09-17T23:38:40.790 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:38:40.788+0000 7fc275b74640 -1 reset not still connected to 0x559cc0696ea0 2024-09-17T23:38:40.790 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:38:40.788+0000 7fc275b74640 -1 reset not still connected to 0x559cc0697c70 2024-09-17T23:38:40.790 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:38:40.788+0000 7fc275b74640 -1 reset not still connected to 0x559cc0ee6ea0 2024-09-17T23:38:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:38:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:38:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:38:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:38:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:38:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:38:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-17T23:38:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-17T23:38:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd9 2024-09-17T23:38:41.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd8 2024-09-17T23:38:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-17T23:38:41.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-17T23:38:41.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:38:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:39:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=86f7d7bdc278729926738d6bfd0aedee 2024-09-17T23:39:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-17T23:39:10.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 86f7d7bdc278729926738d6bfd0aedee != \8\6\f\7\d\7\b\d\c\2\7\8\7\2\9\9\2\6\7\3\8\d\6\b\f\d\0\a\e\d\e\e ]] 2024-09-17T23:39:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=0442f5b9f1bf63878f49d86a5e49d414 2024-09-17T23:39:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-17T23:39:11.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=87bd577da9a3836dc08901ba7ca925d6 2024-09-17T23:39:11.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-17T23:39:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 0442f5b9f1bf63878f49d86a5e49d414 != \0\4\4\2\f\5\b\9\f\1\b\f\6\3\8\7\8\f\4\9\d\8\6\a\5\e\4\9\d\4\1\4 ]] 2024-09-17T23:39:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 87bd577da9a3836dc08901ba7ca925d6 != \8\7\b\d\5\7\7\d\a\9\a\3\8\3\6\d\c\0\8\9\0\1\b\a\7\c\a\9\2\5\d\6 ]] 2024-09-17T23:39:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=b0e848de2e2067a82e01a92e931beb6c 2024-09-17T23:39:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-17T23:39:11.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ b0e848de2e2067a82e01a92e931beb6c != \b\0\e\8\4\8\d\e\2\e\2\0\6\7\a\8\2\e\0\1\a\9\2\e\9\3\1\b\e\b\6\c ]] 2024-09-17T23:39:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=8c0f3f87b3e3a84855298a70649b6707 2024-09-17T23:39:11.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-17T23:39:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 8c0f3f87b3e3a84855298a70649b6707 != \8\c\0\f\3\f\8\7\b\3\e\3\a\8\4\8\5\5\2\9\8\a\7\0\6\4\9\b\6\7\0\7 ]] 2024-09-17T23:39:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=67014364d39bad64dcdda8336332a559 2024-09-17T23:39:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-17T23:39:12.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 67014364d39bad64dcdda8336332a559 != \6\7\0\1\4\3\6\4\d\3\9\b\a\d\6\4\d\c\d\d\a\8\3\3\6\3\3\2\a\5\5\9 ]] 2024-09-17T23:39:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=17787467d690d2a6db747d31b236556f 2024-09-17T23:39:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-17T23:39:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 17787467d690d2a6db747d31b236556f != \1\7\7\8\7\4\6\7\d\6\9\0\d\2\a\6\d\b\7\4\7\d\3\1\b\2\3\6\5\5\6\f ]] 2024-09-17T23:39:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:39:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 44236 2024-09-17T23:39:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:39:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 44238 2024-09-17T23:39:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:39:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 44240 2024-09-17T23:39:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:39:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 44242 2024-09-17T23:39:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:39:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 44244 2024-09-17T23:39:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:39:12.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 44246 2024-09-17T23:39:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=d791c9d536556294e678bf24ff7e9384 2024-09-17T23:39:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-17T23:39:12.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ d791c9d536556294e678bf24ff7e9384 != \d\7\9\1\c\9\d\5\3\6\5\5\6\2\9\4\e\6\7\8\b\f\2\4\f\f\7\e\9\3\8\4 ]] 2024-09-17T23:39:12.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:39:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 44248 2024-09-17T23:39:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:39:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 44250 2024-09-17T23:39:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=b5721ae1244d4b32dc46ddf9e4cefe44 2024-09-17T23:39:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-17T23:39:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ b5721ae1244d4b32dc46ddf9e4cefe44 != \b\5\7\2\1\a\e\1\2\4\4\d\4\b\3\2\d\c\4\6\d\d\f\9\e\4\c\e\f\e\4\4 ]] 2024-09-17T23:39:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=11eaf45cb73cf087d93c9b162e31a867 2024-09-17T23:39:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-17T23:39:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 11eaf45cb73cf087d93c9b162e31a867 != \1\1\e\a\f\4\5\c\b\7\3\c\f\0\8\7\d\9\3\c\9\b\1\6\2\e\3\1\a\8\6\7 ]] 2024-09-17T23:39:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:39:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 44251 2024-09-17T23:39:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-17T23:39:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-17T23:39:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-17T23:39:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:39:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:39:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-17T23:39:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:39:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:39:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-17T23:39:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:39:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-17T23:39:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-17T23:39:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: f30e9a93-a05d-4762-895f-d20592ac2dcb 2024-09-17T23:39:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:39:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63721472.0,"last_snapshot_bytes":63721472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:39:14.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:38:47 2024-09-17T23:39:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:39:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:39:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:39:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:39:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:12' 2024-09-17T23:39:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:39:14.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:39:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:39:14.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63721472.0,"last_snapshot_bytes":63721472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:39:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:39:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:39:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:39:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:39:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:39:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:47' 2024-09-17T23:39:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-17 23:38:47' 2024-09-17T23:39:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:39:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:38:47' 2024-09-17T23:39:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:39:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1f30e9a93-a05d-4762-895f-d20592ac2dcbup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63721472.0,"last_snapshot_bytes":63721472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:38:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:12image-primary104e49171c-1844-47b3-b661-4003a37d9b73up+replayingreplaying, {"bytes_per_second":30447616.0,"bytes_per_snapshot":61628416.0,"last_snapshot_bytes":61628416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:38:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:12image-primary27b92e73e-2de8-435e-8340-c980306d099cup+replayingreplaying, {"bytes_per_second":22888448.0,"bytes_per_snapshot":61468672.0,"last_snapshot_bytes":61468672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:38:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:12image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63787008.0,"last_snapshot_bytes":63787008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:38:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:12image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+replayingreplaying, {"bytes_per_second":36102144.0,"bytes_per_snapshot":72204288.0,"last_snapshot_bytes":72204288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:38:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:12image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+replayingreplaying, {"bytes_per_second":25337856.0,"bytes_per_snapshot":65097728.0,"last_snapshot_bytes":65097728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:38:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:12image-primary69960a41a-881d-4c48-aa63-09f329f03773up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64937984.0,"last_snapshot_bytes":64937984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:38:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:12image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65249280.0,"last_snapshot_bytes":65249280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:38:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:12image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65216512.0,"last_snapshot_bytes":65216512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:38:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:12image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+replayingreplaying, {"bytes_per_second":30349312.0,"bytes_per_snapshot":71651328.0,"last_snapshot_bytes":71651328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:38:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:12' 2024-09-17T23:39:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:39:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-17T23:39:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:38:47' 2024-09-17T23:39:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-17T23:39:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:39:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:39:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-17T23:39:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:39:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:14.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:39:14.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-17T23:39:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-17T23:39:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:39:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-17T23:39:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:39:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:39:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary1 2024-09-17T23:39:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary1 2024-09-17T23:39:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-17T23:39:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-17T23:39:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:39:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:39:16.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-17T23:39:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:39:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:39:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-17T23:39:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:39:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-17T23:39:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-17T23:39:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 7b92e73e-2de8-435e-8340-c980306d099c 2024-09-17T23:39:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:39:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2048955.73,"bytes_per_snapshot":61468672.0,"last_snapshot_bytes":61468672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:39:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:17 2024-09-17T23:39:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:39:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:39:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:39:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:39:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:17' 2024-09-17T23:39:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:39:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:39:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:39:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2048955.73,"bytes_per_snapshot":61468672.0,"last_snapshot_bytes":61468672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:39:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:39:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:39:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:39:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:39:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:39:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-17 23:39:17' 2024-09-17T23:39:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:39:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:39:17' 2024-09-17T23:39:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:17.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:39:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK942634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary104e49171c-1844-47b3-b661-4003a37d9b73up+replayingreplaying, {"bytes_per_second":2054280.53,"bytes_per_snapshot":61628416.0,"last_snapshot_bytes":61628416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17image-primary27b92e73e-2de8-435e-8340-c980306d099cup+replayingreplaying, {"bytes_per_second":2048955.73,"bytes_per_snapshot":61468672.0,"last_snapshot_bytes":61468672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+replayingreplaying, {"bytes_per_second":2126233.6,"bytes_per_snapshot":63787008.0,"last_snapshot_bytes":63787008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+replayingreplaying, {"bytes_per_second":2406809.6,"bytes_per_snapshot":72204288.0,"last_snapshot_bytes":72204288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+replayingreplaying, {"bytes_per_second":2169924.27,"bytes_per_snapshot":65097728.0,"last_snapshot_bytes":65097728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17image-primary69960a41a-881d-4c48-aa63-09f329f03773up+replayingreplaying, {"bytes_per_second":2164599.47,"bytes_per_snapshot":64937984.0,"last_snapshot_bytes":64937984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+replayingreplaying, {"bytes_per_second":2174976.0,"bytes_per_snapshot":65249280.0,"last_snapshot_bytes":65249280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+replayingreplaying, {"bytes_per_second":2173883.73,"bytes_per_snapshot":65216512.0,"last_snapshot_bytes":65216512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+replayingreplaying, {"bytes_per_second":2388377.6,"bytes_per_snapshot":71651328.0,"last_snapshot_bytes":71651328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17' 2024-09-17T23:39:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:39:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-17T23:39:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-17T23:39:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:39:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:39:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:17.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-17T23:39:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:39:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:39:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-17T23:39:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-17T23:39:19.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:39:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-17T23:39:19.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:39:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:39:19.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary2 2024-09-17T23:39:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary2 2024-09-17T23:39:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-17T23:39:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-17T23:39:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:39:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:39:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-17T23:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:39:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-17T23:39:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:39:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:20.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-17T23:39:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-17T23:39:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: e77d98ed-c621-4c06-8624-da95d40e9209 2024-09-17T23:39:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:39:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2126233.6,"bytes_per_snapshot":63787008.0,"last_snapshot_bytes":63787008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:39:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:17 2024-09-17T23:39:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:39:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:39:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:39:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:39:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:17' 2024-09-17T23:39:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:39:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:39:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:39:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2126233.6,"bytes_per_snapshot":63787008.0,"last_snapshot_bytes":63787008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:39:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:39:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:39:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:39:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:39:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:39:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-17 23:39:17' 2024-09-17T23:39:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:39:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:39:17' 2024-09-17T23:39:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:39:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK842634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary104e49171c-1844-47b3-b661-4003a37d9b73up+replayingreplaying, {"bytes_per_second":2054280.53,"bytes_per_snapshot":61628416.0,"last_snapshot_bytes":61628416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17image-primary3e77d98ed-c621-4c06-8624-da95d40e9209up+replayingreplaying, {"bytes_per_second":2126233.6,"bytes_per_snapshot":63787008.0,"last_snapshot_bytes":63787008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+replayingreplaying, {"bytes_per_second":2406809.6,"bytes_per_snapshot":72204288.0,"last_snapshot_bytes":72204288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+replayingreplaying, {"bytes_per_second":2169924.27,"bytes_per_snapshot":65097728.0,"last_snapshot_bytes":65097728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17image-primary69960a41a-881d-4c48-aa63-09f329f03773up+replayingreplaying, {"bytes_per_second":2164599.47,"bytes_per_snapshot":64937984.0,"last_snapshot_bytes":64937984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+replayingreplaying, {"bytes_per_second":2174976.0,"bytes_per_snapshot":65249280.0,"last_snapshot_bytes":65249280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+replayingreplaying, {"bytes_per_second":2173883.73,"bytes_per_snapshot":65216512.0,"last_snapshot_bytes":65216512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+replayingreplaying, {"bytes_per_second":2388377.6,"bytes_per_snapshot":71651328.0,"last_snapshot_bytes":71651328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:17' 2024-09-17T23:39:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:39:20.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-17T23:39:20.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-17T23:39:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:39:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:39:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-17T23:39:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:39:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:39:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-17T23:39:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-17T23:39:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:39:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-17T23:39:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:39:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:39:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary3 2024-09-17T23:39:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary3 2024-09-17T23:39:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-17T23:39:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-17T23:39:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:39:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:39:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-17T23:39:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:39:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:39:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-17T23:39:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:23.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:39:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:23.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-17T23:39:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-17T23:39:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 4b621ebc-252c-483b-87b9-ccdacbc72bd1 2024-09-17T23:39:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:39:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2406809.6,"bytes_per_snapshot":72204288.0,"last_snapshot_bytes":72204288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:39:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:17 2024-09-17T23:39:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:39:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:39:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:39:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:39:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:22' 2024-09-17T23:39:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:39:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:39:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:39:23.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2406809.6,"bytes_per_snapshot":72204288.0,"last_snapshot_bytes":72204288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:39:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:39:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:39:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:39:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:39:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:39:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-17 23:39:17' 2024-09-17T23:39:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:39:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:39:17' 2024-09-17T23:39:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:39:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK742634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary104e49171c-1844-47b3-b661-4003a37d9b73up+replayingreplaying, {"bytes_per_second":2054280.53,"bytes_per_snapshot":61628416.0,"last_snapshot_bytes":61628416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:22image-primary44b621ebc-252c-483b-87b9-ccdacbc72bd1up+replayingreplaying, {"bytes_per_second":2406809.6,"bytes_per_snapshot":72204288.0,"last_snapshot_bytes":72204288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:22image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+replayingreplaying, {"bytes_per_second":2169924.27,"bytes_per_snapshot":65097728.0,"last_snapshot_bytes":65097728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:22image-primary69960a41a-881d-4c48-aa63-09f329f03773up+replayingreplaying, {"bytes_per_second":2164599.47,"bytes_per_snapshot":64937984.0,"last_snapshot_bytes":64937984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:22image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+replayingreplaying, {"bytes_per_second":2174976.0,"bytes_per_snapshot":65249280.0,"last_snapshot_bytes":65249280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:22image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+replayingreplaying, {"bytes_per_second":2173883.73,"bytes_per_snapshot":65216512.0,"last_snapshot_bytes":65216512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:22image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+replayingreplaying, {"bytes_per_second":2388377.6,"bytes_per_snapshot":71651328.0,"last_snapshot_bytes":71651328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:22' 2024-09-17T23:39:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:39:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-17T23:39:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-17T23:39:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:39:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:39:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-17T23:39:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:39:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:39:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-17T23:39:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-17T23:39:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:39:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-17T23:39:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:39:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:39:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary4 2024-09-17T23:39:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary4 2024-09-17T23:39:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-17T23:39:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-17T23:39:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:39:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:39:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-17T23:39:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:39:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:39:26.359 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:39:26.357+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0a64eea0 2024-09-17T23:39:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-17T23:39:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:39:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-17T23:39:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-17T23:39:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 224b1b66-617c-4ba4-b6f3-1c7207f1fadc 2024-09-17T23:39:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:39:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2169924.27,"bytes_per_snapshot":65097728.0,"last_snapshot_bytes":65097728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:39:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:17 2024-09-17T23:39:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:39:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:39:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:39:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:39:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:22' 2024-09-17T23:39:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:39:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:39:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:39:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2169924.27,"bytes_per_snapshot":65097728.0,"last_snapshot_bytes":65097728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:39:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:39:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:39:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:39:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:39:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:39:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-17 23:39:17' 2024-09-17T23:39:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:39:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:39:17' 2024-09-17T23:39:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:39:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK642634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary104e49171c-1844-47b3-b661-4003a37d9b73up+replayingreplaying, {"bytes_per_second":2054280.53,"bytes_per_snapshot":61628416.0,"last_snapshot_bytes":61628416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:22image-primary5224b1b66-617c-4ba4-b6f3-1c7207f1fadcup+replayingreplaying, {"bytes_per_second":2169924.27,"bytes_per_snapshot":65097728.0,"last_snapshot_bytes":65097728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:22image-primary69960a41a-881d-4c48-aa63-09f329f03773up+replayingreplaying, {"bytes_per_second":2164599.47,"bytes_per_snapshot":64937984.0,"last_snapshot_bytes":64937984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:22image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+replayingreplaying, {"bytes_per_second":2174976.0,"bytes_per_snapshot":65249280.0,"last_snapshot_bytes":65249280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:22image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+replayingreplaying, {"bytes_per_second":2173883.73,"bytes_per_snapshot":65216512.0,"last_snapshot_bytes":65216512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:22image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+replayingreplaying, {"bytes_per_second":2388377.6,"bytes_per_snapshot":71651328.0,"last_snapshot_bytes":71651328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:22' 2024-09-17T23:39:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:39:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-17T23:39:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-17T23:39:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:39:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:39:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:26.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-17T23:39:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:39:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:39:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-17T23:39:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-17T23:39:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:39:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-17T23:39:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:39:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:39:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary5 2024-09-17T23:39:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary5 2024-09-17T23:39:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-17T23:39:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-17T23:39:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:39:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:39:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-17T23:39:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:39:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:28.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:39:29.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-17T23:39:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:39:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-17T23:39:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-17T23:39:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9960a41a-881d-4c48-aa63-09f329f03773 2024-09-17T23:39:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:39:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2164599.47,"bytes_per_snapshot":64937984.0,"last_snapshot_bytes":64937984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:39:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:17 2024-09-17T23:39:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:39:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:39:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:39:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:39:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:27' 2024-09-17T23:39:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:39:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:39:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:39:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2164599.47,"bytes_per_snapshot":64937984.0,"last_snapshot_bytes":64937984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:39:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:39:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:39:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:39:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:39:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:39:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-17 23:39:17' 2024-09-17T23:39:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:39:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:39:17' 2024-09-17T23:39:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:39:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK542634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary104e49171c-1844-47b3-b661-4003a37d9b73up+replayingreplaying, {"bytes_per_second":2054280.53,"bytes_per_snapshot":61628416.0,"last_snapshot_bytes":61628416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:27image-primary69960a41a-881d-4c48-aa63-09f329f03773up+replayingreplaying, {"bytes_per_second":2164599.47,"bytes_per_snapshot":64937984.0,"last_snapshot_bytes":64937984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:27image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+replayingreplaying, {"bytes_per_second":2174976.0,"bytes_per_snapshot":65249280.0,"last_snapshot_bytes":65249280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:27image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+replayingreplaying, {"bytes_per_second":2173883.73,"bytes_per_snapshot":65216512.0,"last_snapshot_bytes":65216512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:27image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+replayingreplaying, {"bytes_per_second":2388377.6,"bytes_per_snapshot":71651328.0,"last_snapshot_bytes":71651328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:27' 2024-09-17T23:39:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:39:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-17T23:39:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-17T23:39:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:39:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:39:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-17T23:39:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:39:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:39:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-17T23:39:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-17T23:39:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:39:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-17T23:39:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:39:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:39:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary6 2024-09-17T23:39:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary6 2024-09-17T23:39:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-17T23:39:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-17T23:39:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:39:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:39:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-17T23:39:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:39:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:39:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-17T23:39:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:39:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:32.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-17T23:39:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-17T23:39:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 3e92b0f8-d381-4971-92c7-09af435faee0 2024-09-17T23:39:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:39:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2174976.0,"bytes_per_snapshot":65249280.0,"last_snapshot_bytes":65249280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"} 2024-09-17T23:39:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:39:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:17 2024-09-17T23:39:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:39:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:39:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:39:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:39:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:32' 2024-09-17T23:39:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:39:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:39:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:39:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2174976.0,"bytes_per_snapshot":65249280.0,"last_snapshot_bytes":65249280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"} 2024-09-17T23:39:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:39:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:39:32.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:32.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:39:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:39:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:39:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:39:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-17 23:39:17' 2024-09-17T23:39:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:39:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:39:17' 2024-09-17T23:39:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:39:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary104e49171c-1844-47b3-b661-4003a37d9b73up+replayingreplaying, {"bytes_per_second":2054280.53,"bytes_per_snapshot":61628416.0,"last_snapshot_bytes":61628416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:32image-primary73e92b0f8-d381-4971-92c7-09af435faee0up+replayingreplaying, {"bytes_per_second":2174976.0,"bytes_per_snapshot":65249280.0,"last_snapshot_bytes":65249280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:32image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+replayingreplaying, {"bytes_per_second":2173883.73,"bytes_per_snapshot":65216512.0,"last_snapshot_bytes":65216512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:32image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+replayingreplaying, {"bytes_per_second":2388377.6,"bytes_per_snapshot":71651328.0,"last_snapshot_bytes":71651328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:32' 2024-09-17T23:39:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:39:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-17T23:39:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-17T23:39:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:39:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:39:32.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-17T23:39:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:39:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:39:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-17T23:39:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-17T23:39:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:39:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-17T23:39:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:39:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:39:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary7 2024-09-17T23:39:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary7 2024-09-17T23:39:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-17T23:39:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-17T23:39:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:39:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:39:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-17T23:39:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:39:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:39:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-17T23:39:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:39:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:35.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:35.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-17T23:39:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-17T23:39:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a6f40b0d-defd-4e24-86c9-89901edafb97 2024-09-17T23:39:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:39:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2173883.73,"bytes_per_snapshot":65216512.0,"last_snapshot_bytes":65216512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:39:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:17 2024-09-17T23:39:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:39:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:39:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:39:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:39:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:32' 2024-09-17T23:39:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:39:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:39:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:39:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2173883.73,"bytes_per_snapshot":65216512.0,"last_snapshot_bytes":65216512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:39:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:39:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:39:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:39:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:39:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:39:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-17 23:39:17' 2024-09-17T23:39:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:39:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:39:17' 2024-09-17T23:39:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:39:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK342634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary104e49171c-1844-47b3-b661-4003a37d9b73up+replayingreplaying, {"bytes_per_second":2054280.53,"bytes_per_snapshot":61628416.0,"last_snapshot_bytes":61628416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:32image-primary8a6f40b0d-defd-4e24-86c9-89901edafb97up+replayingreplaying, {"bytes_per_second":2173883.73,"bytes_per_snapshot":65216512.0,"last_snapshot_bytes":65216512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:32image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+replayingreplaying, {"bytes_per_second":2388377.6,"bytes_per_snapshot":71651328.0,"last_snapshot_bytes":71651328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:32' 2024-09-17T23:39:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:39:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-17T23:39:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-17T23:39:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:39:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:39:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-17T23:39:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:39:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:39:35.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-17T23:39:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-17T23:39:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:39:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-17T23:39:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:39:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:39:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary8 2024-09-17T23:39:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary8 2024-09-17T23:39:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-17T23:39:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-17T23:39:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:39:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:39:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-17T23:39:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:39:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:39:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-17T23:39:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:39:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:38.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-17T23:39:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-17T23:39:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 913c3aca-74b9-4a86-9309-65b773ebf0ba 2024-09-17T23:39:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:39:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2388377.6,"bytes_per_snapshot":71651328.0,"last_snapshot_bytes":71651328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"} 2024-09-17T23:39:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:39:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:17 2024-09-17T23:39:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:39:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:39:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:39:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:39:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:37' 2024-09-17T23:39:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:39:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:39:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:39:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2388377.6,"bytes_per_snapshot":71651328.0,"last_snapshot_bytes":71651328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"} 2024-09-17T23:39:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:39:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:39:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:39:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:39:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:39:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:39:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-17 23:39:17' 2024-09-17T23:39:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:38.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:39:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:39:17' 2024-09-17T23:39:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:38.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:39:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK242634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary104e49171c-1844-47b3-b661-4003a37d9b73up+replayingreplaying, {"bytes_per_second":2054280.53,"bytes_per_snapshot":61628416.0,"last_snapshot_bytes":61628416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:37image-primary9913c3aca-74b9-4a86-9309-65b773ebf0baup+replayingreplaying, {"bytes_per_second":2388377.6,"bytes_per_snapshot":71651328.0,"last_snapshot_bytes":71651328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616321,"remote_snapshot_timestamp":1726616321,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:37' 2024-09-17T23:39:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:39:38.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-17T23:39:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:38.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-17T23:39:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:39:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:38.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:39:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-17T23:39:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:39:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:39:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-17T23:39:38.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-17T23:39:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:39:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-17T23:39:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:39:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:39:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary9 2024-09-17T23:39:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary9 2024-09-17T23:39:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-17T23:39:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-17T23:39:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:39:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:39:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-17T23:39:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:39:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:39:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-17T23:39:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:41.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:39:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:39:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:39:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-17T23:39:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-17T23:39:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 4e49171c-1844-47b3-b661-4003a37d9b73 2024-09-17T23:39:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:39:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2054280.53,"bytes_per_snapshot":61628416.0,"last_snapshot_bytes":61628416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:39:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:17 2024-09-17T23:39:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:39:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:39:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:39:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:39:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:39:37' 2024-09-17T23:39:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:39:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:39:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:39:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2054280.53,"bytes_per_snapshot":61628416.0,"last_snapshot_bytes":61628416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"} 2024-09-17T23:39:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:39:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:39:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:39:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:39:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:39:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:39:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-17 23:39:17' 2024-09-17T23:39:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:39:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:39:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:39:17' 2024-09-17T23:39:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:39:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:39:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:39:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary104e49171c-1844-47b3-b661-4003a37d9b73up+replayingreplaying, {"bytes_per_second":2054280.53,"bytes_per_snapshot":61628416.0,"last_snapshot_bytes":61628416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616320,"remote_snapshot_timestamp":1726616320,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:39:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:39:37' 2024-09-17T23:39:41.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:39:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-17T23:39:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:39:17' 2024-09-17T23:39:41.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-17T23:39:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:39:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:39:41.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:39:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:39:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-17T23:39:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:39:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:39:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-17T23:39:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-17T23:39:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:39:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-17T23:39:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:39:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:39:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:39:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary10 2024-09-17T23:39:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary10 2024-09-17T23:39:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..10} 2024-09-17T23:39:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS=() 2024-09-17T23:39:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS=() 2024-09-17T23:39:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS=() 2024-09-17T23:39:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS=() 2024-09-17T23:39:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ RET=0 2024-09-17T23:39:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-17T23:39:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary1 2024-09-17T23:39:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-17T23:39:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:39:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:39:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-17T23:39:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:39:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:39:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:39:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:39:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-17T23:39:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:39:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:39:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-17T23:39:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:39:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-17T23:39:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:39:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-17T23:39:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary1': 2024-09-17T23:39:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:39:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:39:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:39:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1532e29119cd 2024-09-17T23:39:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1532e29119cd 2024-09-17T23:39:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:39:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:39:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:39:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:39:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:39:43 2024 2024-09-17T23:39:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:39:43 2024 2024-09-17T23:39:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:39:43 2024 2024-09-17T23:39:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:39:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:39:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8a8eecbf-6b3c-4a59-8cac-b25382616701 2024-09-17T23:39:44.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:39:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:39:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:39:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-17T23:39:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd0 2024-09-17T23:39:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:39:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-17T23:39:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:39:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:39:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:39:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: ed2a9c98-9447-468a-92cb-03ad0c8d990c 2024-09-17T23:39:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:39:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:39:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:39:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:39:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:39:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:39:44.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary1 2024-09-17T23:39:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-17T23:39:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:39:44.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary1 2024-09-17T23:39:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:39:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-17T23:39:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:39:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:39:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-17T23:39:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:39:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary2 2024-09-17T23:39:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary1 2024-09-17T23:39:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-17T23:39:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:39:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:39:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:39:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary1 2024-09-17T23:39:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:39:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-17T23:39:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-17T23:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-17T23:39:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:39:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:39:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-17T23:39:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:39:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-17T23:39:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:39:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-17T23:39:45.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:39:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 335 2024-09-17T23:39:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-17T23:39:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary2': 2024-09-17T23:39:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:39:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:39:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:39:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1542ccd9dffa 2024-09-17T23:39:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1542ccd9dffa 2024-09-17T23:39:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:39:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:39:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:39:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:39:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:39:44 2024 2024-09-17T23:39:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:39:44 2024 2024-09-17T23:39:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:39:44 2024 2024-09-17T23:39:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:39:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:39:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 5d47f760-049a-45f6-b08b-cc829d2fd297 2024-09-17T23:39:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:39:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:39:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:39:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-17T23:39:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd1 2024-09-17T23:39:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:39:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-17T23:39:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:39:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:39:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:39:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 89a18252-29e6-4a72-8cc8-fee4e3d95a2f 2024-09-17T23:39:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:39:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:39:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:39:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:39:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:39:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:39:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:45.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary2 2024-09-17T23:39:45.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-17T23:39:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:39:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary2 2024-09-17T23:39:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:39:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-17T23:39:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-17T23:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary3 2024-09-17T23:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-17T23:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:39:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary2 2024-09-17T23:39:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:39:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:39:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-17T23:39:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary2 2024-09-17T23:39:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:39:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:39:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:39:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:39:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-17T23:39:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-17T23:39:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:39:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:39:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-17T23:39:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:39:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-17T23:39:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:39:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-17T23:39:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:39:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-17T23:39:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 337 2024-09-17T23:39:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary3': 2024-09-17T23:39:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:39:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:39:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:39:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 155157c9deb9 2024-09-17T23:39:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.155157c9deb9 2024-09-17T23:39:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:39:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:39:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:39:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:39:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:39:45 2024 2024-09-17T23:39:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:39:45 2024 2024-09-17T23:39:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:39:45 2024 2024-09-17T23:39:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:39:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:39:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: dc1f808f-0bc4-4dc3-928e-846d00114044 2024-09-17T23:39:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:39:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:39:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:39:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-17T23:39:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd2 2024-09-17T23:39:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:39:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-17T23:39:46.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:39:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:39:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:39:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 6a1390d4-c3ce-4eb5-9680-344376bbd393 2024-09-17T23:39:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:39:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:39:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:39:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:39:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:39:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:39:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary3 2024-09-17T23:39:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-17T23:39:46.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:39:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary3 2024-09-17T23:39:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:39:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-17T23:39:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:39:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:39:46.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:39:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:39:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:39:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary3 2024-09-17T23:39:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-17T23:39:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary4 2024-09-17T23:39:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-17T23:39:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary3 2024-09-17T23:39:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-17T23:39:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:39:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:39:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-17T23:39:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:39:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:39:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:39:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:39:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-17T23:39:46.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:39:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:39:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-17T23:39:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:39:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-17T23:39:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:39:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-17T23:39:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:39:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 339 2024-09-17T23:39:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-17T23:39:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary4': 2024-09-17T23:39:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:39:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:39:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:39:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 156040f6df61 2024-09-17T23:39:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.156040f6df61 2024-09-17T23:39:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:39:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:39:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:39:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:39:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:39:46 2024 2024-09-17T23:39:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:39:46 2024 2024-09-17T23:39:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:39:46 2024 2024-09-17T23:39:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:39:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:39:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9409f57e-7aad-46f4-8c9e-6fe60e04c0ae 2024-09-17T23:39:47.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:39:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:39:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:39:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-17T23:39:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd3 2024-09-17T23:39:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:39:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-17T23:39:47.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:39:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:39:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:39:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: db9b31db-b472-4245-af29-bdb89f0804b3 2024-09-17T23:39:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:39:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:39:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:39:47.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:39:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:39:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:39:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary4 2024-09-17T23:39:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-17T23:39:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:39:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary4 2024-09-17T23:39:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:39:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-17T23:39:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:39:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:39:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:39:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:39:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:39:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-17T23:39:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary5 2024-09-17T23:39:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-17T23:39:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:39:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:39:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-17T23:39:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:39:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:39:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:39:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:39:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-17T23:39:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary4 2024-09-17T23:39:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary4 2024-09-17T23:39:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-17T23:39:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:39:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:39:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-17T23:39:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:39:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-17T23:39:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:39:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-17T23:39:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:39:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 341 2024-09-17T23:39:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:39:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:39:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:39:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-17T23:39:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:48.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary5': 2024-09-17T23:39:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:39:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:39:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:39:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 156e890dbfb2 2024-09-17T23:39:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.156e890dbfb2 2024-09-17T23:39:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:39:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:39:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:39:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:39:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:39:47 2024 2024-09-17T23:39:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:39:47 2024 2024-09-17T23:39:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:39:47 2024 2024-09-17T23:39:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:39:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:39:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: fd8e79f3-d28a-4074-9d90-015d42d369a6 2024-09-17T23:39:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:39:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:39:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:39:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-17T23:39:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd4 2024-09-17T23:39:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:39:48.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-17T23:39:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:39:48.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:39:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:39:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 2c9e2212-ef2c-49bc-98d9-a1ffd13b50d2 2024-09-17T23:39:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:39:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:39:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:39:48.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:39:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:39:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:39:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary5 2024-09-17T23:39:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-17T23:39:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:39:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary5 2024-09-17T23:39:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:39:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-17T23:39:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary5 2024-09-17T23:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-17T23:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary6 2024-09-17T23:39:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-17T23:39:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary5 2024-09-17T23:39:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-17T23:39:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:39:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:39:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-17T23:39:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:39:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:39:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:39:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:39:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-17T23:39:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:39:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:39:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-17T23:39:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:39:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-17T23:39:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:39:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-17T23:39:49.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:39:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 344 2024-09-17T23:39:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-17T23:39:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:39:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:39:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:39:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary6': 2024-09-17T23:39:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:39:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:39:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:39:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1580f43082fd 2024-09-17T23:39:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1580f43082fd 2024-09-17T23:39:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:39:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:39:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:39:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:39:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:39:48 2024 2024-09-17T23:39:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:39:48 2024 2024-09-17T23:39:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:39:48 2024 2024-09-17T23:39:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:39:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:39:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 83c782ac-6390-4046-a744-fa055dd473a9 2024-09-17T23:39:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:39:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:39:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:39:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-17T23:39:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd5 2024-09-17T23:39:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:39:49.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-17T23:39:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:39:49.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:39:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:39:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: d8ca4f50-1825-42f2-89be-87c9068a2be5 2024-09-17T23:39:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:39:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:39:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:39:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:39:49.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:39:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:39:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary6 2024-09-17T23:39:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-17T23:39:49.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:39:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary6 2024-09-17T23:39:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:39:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-17T23:39:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:39:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:39:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-17T23:39:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary7 2024-09-17T23:39:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:39:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-17T23:39:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:39:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:39:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:39:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:39:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-17T23:39:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:39:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:39:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:39:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:39:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-17T23:39:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary6 2024-09-17T23:39:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary6 2024-09-17T23:39:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-17T23:39:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:39:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:39:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-17T23:39:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:39:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-17T23:39:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:39:49.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-17T23:39:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 343 2024-09-17T23:39:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:39:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:39:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:39:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:39:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 347 2024-09-17T23:39:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-17T23:39:50.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary7': 2024-09-17T23:39:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:39:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:39:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:39:50.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1591679e8414 2024-09-17T23:39:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1591679e8414 2024-09-17T23:39:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:39:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:39:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:39:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:39:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:39:49 2024 2024-09-17T23:39:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:39:49 2024 2024-09-17T23:39:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:39:49 2024 2024-09-17T23:39:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:39:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:39:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 2683111f-ced2-4294-a3f5-061efd582a2c 2024-09-17T23:39:50.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:39:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:39:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:39:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-17T23:39:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd6 2024-09-17T23:39:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:39:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-17T23:39:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:39:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:39:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:39:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: f0a3af62-4f1d-4532-8f78-0ac64a735100 2024-09-17T23:39:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:39:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:39:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:39:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:39:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:39:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:39:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary7 2024-09-17T23:39:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-17T23:39:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:39:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary7 2024-09-17T23:39:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:39:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-17T23:39:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:39:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:39:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:39:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:39:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:39:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary7 2024-09-17T23:39:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-17T23:39:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary8 2024-09-17T23:39:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-17T23:39:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary7 2024-09-17T23:39:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-17T23:39:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:39:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:39:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-17T23:39:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:39:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:39:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:39:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:39:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:50.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-17T23:39:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:39:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:39:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-17T23:39:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:39:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-17T23:39:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:39:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-17T23:39:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 346 2024-09-17T23:39:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:39:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:39:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:39:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:39:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 350 2024-09-17T23:39:51.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-17T23:39:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary8': 2024-09-17T23:39:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:39:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:39:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:39:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 15a38bad3b8e 2024-09-17T23:39:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.15a38bad3b8e 2024-09-17T23:39:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:39:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:39:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:39:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:39:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:39:50 2024 2024-09-17T23:39:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:39:50 2024 2024-09-17T23:39:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:39:50 2024 2024-09-17T23:39:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:39:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:39:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 7d1adc6c-e30d-475c-ba6c-ffbffadfed3a 2024-09-17T23:39:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:39:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:39:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:39:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-17T23:39:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd7 2024-09-17T23:39:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:39:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-17T23:39:51.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:39:51.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:39:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:39:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 19ab2928-2718-4f01-b898-938c416eaf4c 2024-09-17T23:39:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:39:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:39:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:39:51.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:39:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:39:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:39:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary8 2024-09-17T23:39:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-17T23:39:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:39:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary8 2024-09-17T23:39:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:39:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-17T23:39:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:39:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:39:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:39:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-17T23:39:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary9 2024-09-17T23:39:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:39:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-17T23:39:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:39:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:39:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:39:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-17T23:39:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary8 2024-09-17T23:39:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:39:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:39:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:39:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:39:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-17T23:39:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary8 2024-09-17T23:39:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-17T23:39:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:39:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:39:51.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-17T23:39:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:39:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:51.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-17T23:39:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:39:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-17T23:39:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 349 2024-09-17T23:39:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:39:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:39:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:39:52.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:39:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 353 2024-09-17T23:39:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-17T23:39:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary9': 2024-09-17T23:39:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:39:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:39:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:39:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 15b4df255fce 2024-09-17T23:39:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.15b4df255fce 2024-09-17T23:39:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:39:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:39:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:39:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:39:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:39:51 2024 2024-09-17T23:39:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:39:51 2024 2024-09-17T23:39:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:39:51 2024 2024-09-17T23:39:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:39:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:39:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 3c802c28-d835-428e-9487-2297aed8b734 2024-09-17T23:39:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:39:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:39:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:39:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-17T23:39:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd8 2024-09-17T23:39:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:39:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-17T23:39:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:39:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:39:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:39:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 07765241-0929-4dd7-8c0c-76b4d5382f0b 2024-09-17T23:39:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:39:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:39:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:39:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:39:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:39:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:39:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary9 2024-09-17T23:39:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-17T23:39:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:39:52.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary9 2024-09-17T23:39:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:39:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-17T23:39:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:39:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:39:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:39:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:39:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-17T23:39:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary10 2024-09-17T23:39:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary9 2024-09-17T23:39:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-17T23:39:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:39:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:39:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:39:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary9 2024-09-17T23:39:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:39:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-17T23:39:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-17T23:39:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:39:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:52.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:39:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:39:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:39:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:39:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-17T23:39:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:39:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:39:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:39:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-17T23:39:52.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:39:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:39:52.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-17T23:39:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:39:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-17T23:39:53.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 352 2024-09-17T23:39:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:39:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:39:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:39:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:39:53.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 356 2024-09-17T23:39:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:39:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:39:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:39:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-17T23:39:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary10': 2024-09-17T23:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 15c7ac31403c 2024-09-17T23:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.15c7ac31403c 2024-09-17T23:39:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:39:52 2024 2024-09-17T23:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:39:52 2024 2024-09-17T23:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:39:52 2024 2024-09-17T23:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9b073ae2-3e3a-4e6c-af00-5c64a0ab7d7d 2024-09-17T23:39:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:39:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:39:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:39:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-17T23:39:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd9 2024-09-17T23:39:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:39:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-17T23:39:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:39:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:39:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:39:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: a3cee280-77ad-4232-9a22-5e22b81f8fa4 2024-09-17T23:39:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:39:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:39:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:39:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:39:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:39:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:39:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:39:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary10 2024-09-17T23:39:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-17T23:39:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:39:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary10 2024-09-17T23:39:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:39:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-17T23:39:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:39:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:39:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:39:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:39:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 47768 2024-09-17T23:39:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary10 2024-09-17T23:39:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:39:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:39:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary10 2024-09-17T23:39:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-17T23:39:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:39:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-17T23:39:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 355 2024-09-17T23:39:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 360 2024-09-17T23:39:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:39:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:39:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:39:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:39:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:39:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 359 2024-09-17T23:39:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 358 2024-09-17T23:39:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:39:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:39:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:39:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:39:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:39:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:39:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 361 2024-09-17T23:39:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 362 2024-09-17T23:39:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:39:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:39:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:39:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:56.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:39:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:39:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:39:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:39:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:39:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:39:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 364 2024-09-17T23:39:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:39:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:39:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:39:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 363 2024-09-17T23:39:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 365 2024-09-17T23:39:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 366 2024-09-17T23:39:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:39:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:39:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:39:58.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:39:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:39:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:39:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 367 2024-09-17T23:39:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 368 2024-09-17T23:39:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:39:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:39:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:39:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:39:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:39:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:39:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:39:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:39:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:39:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:40:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 370 2024-09-17T23:40:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 369 2024-09-17T23:40:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:40:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:40:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:40:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:00.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:40:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:40:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:40:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 371 2024-09-17T23:40:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:40:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:40:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:40:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:40:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:40:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:40:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:40:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:40:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:40:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:40:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:40:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:40:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 373 2024-09-17T23:40:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 372 2024-09-17T23:40:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 374 2024-09-17T23:40:02.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 375 2024-09-17T23:40:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 376 2024-09-17T23:40:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:40:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:40:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:40:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:40:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:40:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:40:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 378 2024-09-17T23:40:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:40:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:40:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:40:05.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 379 2024-09-17T23:40:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 377 2024-09-17T23:40:05.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:05.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:05.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:40:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:40:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:40:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:40:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:40:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:40:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:40:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:40:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:40:06.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 380 2024-09-17T23:40:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 381 2024-09-17T23:40:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 384 2024-09-17T23:40:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:40:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:40:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:40:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:40:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:40:06.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:40:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 382 2024-09-17T23:40:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 383 2024-09-17T23:40:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:40:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:40:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:40:07.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 386 2024-09-17T23:40:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 385 2024-09-17T23:40:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:40:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:40:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:40:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:40:08.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:08.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:08.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:40:08.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:40:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 387 2024-09-17T23:40:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 389 2024-09-17T23:40:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:40:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:40:09.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:40:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:40:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:40:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:40:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:40:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:40:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:40:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:40:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:40:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:40:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 388 2024-09-17T23:40:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:40:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:40:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:40:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 390 2024-09-17T23:40:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:40:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:40:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:40:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:40:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:40:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:40:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 391 2024-09-17T23:40:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 392 2024-09-17T23:40:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:40:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:40:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:40:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 394 2024-09-17T23:40:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 393 2024-09-17T23:40:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:40:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:40:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:40:12.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 396 2024-09-17T23:40:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 395 2024-09-17T23:40:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:40:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:40:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:40:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:40:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:40:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:40:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 398 2024-09-17T23:40:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 397 2024-09-17T23:40:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:40:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:40:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:40:14.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:40:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:40:14.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:40:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 399 2024-09-17T23:40:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 400 2024-09-17T23:40:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:40:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:40:15.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:40:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:40:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:40:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:40:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 402 2024-09-17T23:40:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 401 2024-09-17T23:40:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:40:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:40:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:40:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:40:16.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:40:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:40:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:40:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:40:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:40:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:40:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:40:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:40:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 403 2024-09-17T23:40:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 404 2024-09-17T23:40:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 405 2024-09-17T23:40:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 406 2024-09-17T23:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:40:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:40:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:40:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:40:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:40:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:19.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 408 2024-09-17T23:40:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:40:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:40:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:40:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:40:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:40:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:40:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 407 2024-09-17T23:40:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 409 2024-09-17T23:40:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 410 2024-09-17T23:40:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:40:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:40:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:40:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:40:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:40:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:40:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 412 2024-09-17T23:40:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:40:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:40:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:40:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:40:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:40:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:40:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:40:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:40:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:40:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 414 2024-09-17T23:40:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 413 2024-09-17T23:40:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 411 2024-09-17T23:40:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:40:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:40:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:40:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:40:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:40:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:40:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:40:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:40:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:40:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 416 2024-09-17T23:40:23.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 415 2024-09-17T23:40:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:23.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 417 2024-09-17T23:40:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 420 2024-09-17T23:40:24.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:40:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:40:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:40:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 418 2024-09-17T23:40:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 419 2024-09-17T23:40:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:40:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:40:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:40:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:40:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:40:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:40:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:40:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:40:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 421 2024-09-17T23:40:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:26.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 424 2024-09-17T23:40:26.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:40:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:40:26.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:40:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:40:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:40:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:40:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:40:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:40:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:40:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 422 2024-09-17T23:40:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:40:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:40:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:40:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:28.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 423 2024-09-17T23:40:28.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 425 2024-09-17T23:40:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:40:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:40:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:40:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:40:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:40:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:40:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 426 2024-09-17T23:40:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 428 2024-09-17T23:40:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:40:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:40:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:40:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:40:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:40:29.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:40:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 430 2024-09-17T23:40:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:40:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:40:30.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:40:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 429 2024-09-17T23:40:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 427 2024-09-17T23:40:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 432 2024-09-17T23:40:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 431 2024-09-17T23:40:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:40:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:40:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:40:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:40:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:40:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:40:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:40:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:40:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:40:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:40:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:32.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:32.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:40:32.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:40:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 433 2024-09-17T23:40:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 434 2024-09-17T23:40:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 436 2024-09-17T23:40:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 435 2024-09-17T23:40:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:40:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:40:33.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:40:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:40:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:40:33.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:40:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:40:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:40:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:40:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 437 2024-09-17T23:40:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 440 2024-09-17T23:40:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:40:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:34.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:40:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:40:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:40:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:40:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:40:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 439 2024-09-17T23:40:35.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 438 2024-09-17T23:40:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:40:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:40:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:40:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:40:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:40:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:40:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:36.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 441 2024-09-17T23:40:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 442 2024-09-17T23:40:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:40:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:40:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:40:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:40:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:40:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:40:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:36.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:40:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:40:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:40:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:40:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:40:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:40:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 443 2024-09-17T23:40:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 444 2024-09-17T23:40:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:37.429 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:40:37.426+0000 7f706e974640 -1 reset not still connected to 0x55df1d2d3790 2024-09-17T23:40:37.429 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:40:37.426+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4270 2024-09-17T23:40:37.429 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:40:37.426+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4340 2024-09-17T23:40:37.429 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:40:37.426+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb52b0 2024-09-17T23:40:37.429 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:40:37.426+0000 7f706e974640 -1 reset not still connected to 0x55df1ee0bd40 2024-09-17T23:40:37.429 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:40:37.426+0000 7f706e974640 -1 reset not still connected to 0x55df1f36dd40 2024-09-17T23:40:37.429 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:40:37.426+0000 7f706e974640 -1 reset not still connected to 0x55df28212000 2024-09-17T23:40:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 446 2024-09-17T23:40:38.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 445 2024-09-17T23:40:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:40:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:40:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:40:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:40:39.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 447 2024-09-17T23:40:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:40:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:40:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:40:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:40:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:40:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:40:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 450 2024-09-17T23:40:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 448 2024-09-17T23:40:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 449 2024-09-17T23:40:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:40:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:40:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:40:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:40:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:40:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:40:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:40.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:40:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:40:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:40:41.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 451 2024-09-17T23:40:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:40:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:40:41.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:40:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:40:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:40:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:40:42.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 454 2024-09-17T23:40:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 452 2024-09-17T23:40:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 453 2024-09-17T23:40:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:42.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:42.547 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:40:42.545+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4270 2024-09-17T23:40:42.548 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:40:42.546+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4340 2024-09-17T23:40:42.548 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:40:42.546+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb52b0 2024-09-17T23:40:42.548 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:40:42.546+0000 7f706e974640 -1 reset not still connected to 0x55df1f36dd40 2024-09-17T23:40:42.549 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:40:42.546+0000 7f706e974640 -1 reset not still connected to 0x55df202dd040 2024-09-17T23:40:42.549 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:40:42.546+0000 7f706e974640 -1 reset not still connected to 0x55df28212000 2024-09-17T23:40:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 456 2024-09-17T23:40:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 455 2024-09-17T23:40:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:40:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:40:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:40:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:40:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:40:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:40:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:40:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:40:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:40:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:44.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 457 2024-09-17T23:40:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 460 2024-09-17T23:40:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:40:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:40:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:40:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 459 2024-09-17T23:40:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:40:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:40:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:40:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:40:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:40:45.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:40:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:40:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:40:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:40:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 458 2024-09-17T23:40:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 461 2024-09-17T23:40:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 464 2024-09-17T23:40:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:40:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:40:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:40:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:40:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:40:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:40:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:40:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:40:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:40:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 463 2024-09-17T23:40:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 462 2024-09-17T23:40:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:47.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:40:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:40:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:40:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 465 2024-09-17T23:40:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 466 2024-09-17T23:40:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:40:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:40:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:40:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 467 2024-09-17T23:40:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 468 2024-09-17T23:40:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:49.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:49.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:40:49.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:49.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:49.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:40:49.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:40:49.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:40:49.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:40:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:40:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:40:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:40:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:40:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 469 2024-09-17T23:40:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:40:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:40:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:40:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:40:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:40:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:40:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:50.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 470 2024-09-17T23:40:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:51.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 471 2024-09-17T23:40:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 472 2024-09-17T23:40:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:40:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:40:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:40:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:40:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:40:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:40:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:40:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:40:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:40:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 474 2024-09-17T23:40:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 473 2024-09-17T23:40:52.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:40:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:40:52.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:40:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 475 2024-09-17T23:40:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 476 2024-09-17T23:40:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:40:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:40:53.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:40:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:40:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:40:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:40:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 478 2024-09-17T23:40:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 477 2024-09-17T23:40:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:40:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:40:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:40:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:40:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:40:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:40:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 480 2024-09-17T23:40:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 479 2024-09-17T23:40:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:40:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:40:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:40:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:40:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:40:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:40:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 482 2024-09-17T23:40:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 481 2024-09-17T23:40:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:40:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:40:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:40:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:40:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:40:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:40:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:40:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:40:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:40:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:40:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:40:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:40:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 484 2024-09-17T23:40:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 483 2024-09-17T23:40:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:58.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 486 2024-09-17T23:40:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:40:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:40:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:40:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:40:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:40:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:40:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 485 2024-09-17T23:40:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 487 2024-09-17T23:40:59.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:59.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:40:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:40:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:40:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:40:59.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:40:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:40:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:40:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:40:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:40:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:41:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 488 2024-09-17T23:41:00.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 489 2024-09-17T23:41:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 490 2024-09-17T23:41:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:41:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:41:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:41:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:41:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:41:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:41:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:01.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 491 2024-09-17T23:41:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 492 2024-09-17T23:41:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:41:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:41:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:41:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:41:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:41:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:41:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:41:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:41:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:41:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 493 2024-09-17T23:41:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 494 2024-09-17T23:41:02.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:02.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 495 2024-09-17T23:41:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:41:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:41:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:41:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:41:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:41:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:41:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 496 2024-09-17T23:41:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 497 2024-09-17T23:41:04.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 499 2024-09-17T23:41:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:41:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:41:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:41:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:41:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:41:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:41:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 498 2024-09-17T23:41:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 500 2024-09-17T23:41:05.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:41:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:41:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:41:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:41:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:41:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:41:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 502 2024-09-17T23:41:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 501 2024-09-17T23:41:06.326 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:41:06.324+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1e5d5930 2024-09-17T23:41:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:41:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:06.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:41:06.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:41:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:41:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:41:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:41:07.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 503 2024-09-17T23:41:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 504 2024-09-17T23:41:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:41:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:41:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:41:07.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:41:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:41:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:41:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:41:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:41:08.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:41:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 506 2024-09-17T23:41:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:41:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:41:08.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:41:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 505 2024-09-17T23:41:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 507 2024-09-17T23:41:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 508 2024-09-17T23:41:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:41:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:41:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:41:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:41:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:41:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:41:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 509 2024-09-17T23:41:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 510 2024-09-17T23:41:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:41:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:41:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:41:10.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:41:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:41:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:41:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 511 2024-09-17T23:41:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:41:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:41:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:41:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:41:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:41:11.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:41:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:41:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:41:12.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:41:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:41:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:41:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:41:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 512 2024-09-17T23:41:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 514 2024-09-17T23:41:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 513 2024-09-17T23:41:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 516 2024-09-17T23:41:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 515 2024-09-17T23:41:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:41:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:41:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:41:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:41:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:41:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:41:13.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 518 2024-09-17T23:41:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:41:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:41:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:41:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 517 2024-09-17T23:41:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 520 2024-09-17T23:41:15.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 519 2024-09-17T23:41:15.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:41:15.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:41:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:41:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:15.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:41:15.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:15.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:15.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:41:15.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:41:15.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:41:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:41:15.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:41:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 521 2024-09-17T23:41:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:41:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:41:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:41:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:41:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:41:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:41:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:16.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 523 2024-09-17T23:41:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:41:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:41:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:41:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 524 2024-09-17T23:41:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 522 2024-09-17T23:41:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 526 2024-09-17T23:41:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 525 2024-09-17T23:41:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:18.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:41:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:41:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:41:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:41:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:41:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:41:18.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:41:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:41:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:41:18.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 527 2024-09-17T23:41:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 529 2024-09-17T23:41:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:41:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:41:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:41:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:41:19.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:41:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:41:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 530 2024-09-17T23:41:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 528 2024-09-17T23:41:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:41:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:41:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:41:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:41:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:41:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:41:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 531 2024-09-17T23:41:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 532 2024-09-17T23:41:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:41:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:41:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:41:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:41:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:41:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:41:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:41:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:41:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:41:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:41:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:41:22.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:41:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 534 2024-09-17T23:41:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 533 2024-09-17T23:41:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 535 2024-09-17T23:41:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 536 2024-09-17T23:41:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:41:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:41:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:41:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:41:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:23.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:41:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:41:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 537 2024-09-17T23:41:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:41:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:41:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:41:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:41:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:41:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:41:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 539 2024-09-17T23:41:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 538 2024-09-17T23:41:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:41:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:41:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:41:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:41:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:41:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:41:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 541 2024-09-17T23:41:26.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 540 2024-09-17T23:41:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:41:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:41:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:41:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:41:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:41:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:41:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:26.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:26.564 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-17T23:41:26.560+0000 7f0114255640 -1 reset not still connected to 0x557d0a065ee0 2024-09-17T23:41:26.564 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-17T23:41:26.561+0000 7f0114255640 -1 reset not still connected to 0x557d0b464750 2024-09-17T23:41:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:41:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:41:27.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:41:27.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 544 2024-09-17T23:41:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 542 2024-09-17T23:41:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:41:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:28.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:41:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:41:28.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:28.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:41:28.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:28.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:28.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:41:28.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:41:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 543 2024-09-17T23:41:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 545 2024-09-17T23:41:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 546 2024-09-17T23:41:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:29.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 547 2024-09-17T23:41:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 549 2024-09-17T23:41:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:41:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:41:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:41:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:41:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:41:30.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 548 2024-09-17T23:41:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 551 2024-09-17T23:41:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:41:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:41:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:41:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:41:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:41:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:41:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 550 2024-09-17T23:41:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 553 2024-09-17T23:41:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:41:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:41:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:41:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:41:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:41:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:41:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:41:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:41:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:41:32.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 552 2024-09-17T23:41:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:41:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:41:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:41:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 556 2024-09-17T23:41:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:41:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:41:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:41:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 554 2024-09-17T23:41:33.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 555 2024-09-17T23:41:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:41:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:41:33.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:41:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:41:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:41:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:41:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 557 2024-09-17T23:41:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 558 2024-09-17T23:41:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:41:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:41:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:41:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:41:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:41:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:41:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 559 2024-09-17T23:41:35.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:41:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:41:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:41:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:41:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:41:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:41:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 561 2024-09-17T23:41:36.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 560 2024-09-17T23:41:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 562 2024-09-17T23:41:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:41:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:41:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:41:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:41:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:41:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:41:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 563 2024-09-17T23:41:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 564 2024-09-17T23:41:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:41:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:41:37.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 565 2024-09-17T23:41:38.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 566 2024-09-17T23:41:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:41:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:41:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:41:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 568 2024-09-17T23:41:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 567 2024-09-17T23:41:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:41:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:41:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:41:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:41:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:41:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:41:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:41:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:41:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:41:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 569 2024-09-17T23:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:41:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:41:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:41:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:41:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:41:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 570 2024-09-17T23:41:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:41:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:41:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:41:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:41:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:41:41.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:41:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 572 2024-09-17T23:41:41.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 571 2024-09-17T23:41:41.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:41:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:41:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:41:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:41:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:41:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:41:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 573 2024-09-17T23:41:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 574 2024-09-17T23:41:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:43.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 576 2024-09-17T23:41:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 575 2024-09-17T23:41:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:41:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:41:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:41:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:41:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:41:43.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:41:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 577 2024-09-17T23:41:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 578 2024-09-17T23:41:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:41:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:41:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:41:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:41:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:41:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:41:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 579 2024-09-17T23:41:45.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 580 2024-09-17T23:41:45.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:41:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:41:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:41:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:41:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:41:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:41:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 581 2024-09-17T23:41:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 582 2024-09-17T23:41:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:41:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:41:46.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:41:46.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:41:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:41:46.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:41:46.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:41:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:41:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:41:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:41:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:41:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:41:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 583 2024-09-17T23:41:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 584 2024-09-17T23:41:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 585 2024-09-17T23:41:48.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 586 2024-09-17T23:41:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:41:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:41:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:41:48.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:41:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:41:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:41:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 588 2024-09-17T23:41:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:41:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:41:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:41:49.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:49.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:41:49.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:49.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:49.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:41:49.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:41:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 589 2024-09-17T23:41:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 587 2024-09-17T23:41:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 590 2024-09-17T23:41:50.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:50.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:50.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:41:50.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:50.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:50.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:41:50.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:41:50.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:50.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:41:50.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:50.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:50.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:41:50.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:41:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 591 2024-09-17T23:41:51.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 592 2024-09-17T23:41:51.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:41:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:41:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:41:51.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:51.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:41:51.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:51.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:51.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:41:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:41:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:41:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:41:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:41:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 596 2024-09-17T23:41:53.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:41:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:41:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:41:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 593 2024-09-17T23:41:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 594 2024-09-17T23:41:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:41:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:41:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:41:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:41:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:41:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:41:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 595 2024-09-17T23:41:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 597 2024-09-17T23:41:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:54.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 599 2024-09-17T23:41:54.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:41:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:41:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:41:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:41:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:41:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:41:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 598 2024-09-17T23:41:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 600 2024-09-17T23:41:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 601 2024-09-17T23:41:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 602 2024-09-17T23:41:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:56.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:41:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:41:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:41:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:41:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:41:56.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:41:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:41:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:41:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:41:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:41:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:41:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:41:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:41:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:41:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:41:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 605 2024-09-17T23:41:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 604 2024-09-17T23:41:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:41:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:41:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:41:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 606 2024-09-17T23:41:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 603 2024-09-17T23:41:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:41:58.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:58.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:58.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:41:58.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:41:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 607 2024-09-17T23:41:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 608 2024-09-17T23:41:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:41:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:41:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:41:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:41:59.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:59.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:59.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:41:59.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:41:59.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:41:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:41:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:41:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:41:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:41:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:41:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:42:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 609 2024-09-17T23:42:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:42:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:42:00.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:42:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:42:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:42:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:42:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 610 2024-09-17T23:42:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:01.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 612 2024-09-17T23:42:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 611 2024-09-17T23:42:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:42:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:42:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:42:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:42:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:42:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:42:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:42:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:42:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:42:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 613 2024-09-17T23:42:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:42:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:42:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:42:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 614 2024-09-17T23:42:02.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 616 2024-09-17T23:42:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 615 2024-09-17T23:42:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:42:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:42:03.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:42:03.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:03.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:42:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:42:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:42:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 617 2024-09-17T23:42:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:42:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:42:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:42:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:42:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:42:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:42:05.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 619 2024-09-17T23:42:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 618 2024-09-17T23:42:05.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 620 2024-09-17T23:42:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:42:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:42:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:42:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:42:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:42:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:42:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 622 2024-09-17T23:42:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 621 2024-09-17T23:42:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:42:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:42:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:42:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:42:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:42:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:42:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:06.577 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:42:06.574+0000 7fc275b74640 -1 reset not still connected to 0x559cbebc1110 2024-09-17T23:42:06.577 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:42:06.574+0000 7fc275b74640 -1 reset not still connected to 0x559cbebc1520 2024-09-17T23:42:06.577 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:42:06.574+0000 7fc275b74640 -1 reset not still connected to 0x559cbf59fa00 2024-09-17T23:42:06.577 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:42:06.574+0000 7fc275b74640 -1 reset not still connected to 0x559cc4bc20d0 2024-09-17T23:42:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:07.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:42:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:42:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:42:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 624 2024-09-17T23:42:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 623 2024-09-17T23:42:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 626 2024-09-17T23:42:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 625 2024-09-17T23:42:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:42:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:42:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:42:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:42:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:42:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:42:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:42:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:42:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:42:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 627 2024-09-17T23:42:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 629 2024-09-17T23:42:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:09.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:42:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:42:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:42:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:42:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:42:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:42:10.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 628 2024-09-17T23:42:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 630 2024-09-17T23:42:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:42:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:42:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:42:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:42:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:42:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:42:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:10.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 632 2024-09-17T23:42:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 631 2024-09-17T23:42:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:42:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:42:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:42:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:42:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:42:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:42:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:42:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:42:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:42:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:42:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:42:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:42:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 634 2024-09-17T23:42:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 633 2024-09-17T23:42:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:12.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 636 2024-09-17T23:42:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 635 2024-09-17T23:42:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:42:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 47901 2024-09-17T23:42:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:13.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:42:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:42:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:42:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 637 2024-09-17T23:42:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 638 2024-09-17T23:42:14.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:42:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:42:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:42:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:42:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 48041 2024-09-17T23:42:15.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 639 2024-09-17T23:42:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:42:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:42:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:42:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:42:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 48176 2024-09-17T23:42:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 640 2024-09-17T23:42:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:42:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:42:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:42:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:42:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 48328 2024-09-17T23:42:17.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 641 2024-09-17T23:42:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:42:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 48480 2024-09-17T23:42:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:42:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:42:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:42:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:42:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:42:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:42:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 642 2024-09-17T23:42:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:42:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 48633 2024-09-17T23:42:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 643 2024-09-17T23:42:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:42:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 48786 2024-09-17T23:42:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:42:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:42:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 48939 2024-09-17T23:42:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:42:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 49112 2024-09-17T23:42:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-17T23:42:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:42:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 47769 2024-09-17T23:44:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:44:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:44:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:44:45.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 47902 2024-09-17T23:44:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:44:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:44:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:44:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 48042 2024-09-17T23:44:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:44:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:44:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:44:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 48177 2024-09-17T23:44:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:44:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:44:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:44:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 48329 2024-09-17T23:44:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:44:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:44:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:44:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 48481 2024-09-17T23:44:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:44:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:44:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:44:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 48634 2024-09-17T23:44:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:44:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:44:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:44:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 48787 2024-09-17T23:44:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:44:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:44:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:44:51.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 48940 2024-09-17T23:44:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:44:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:44:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:44:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 49113 2024-09-17T23:44:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:44:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-17T23:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 1 2024-09-17T23:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd0 2024-09-17T23:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary1 2024-09-17T23:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary1 2024-09-17T23:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary1 2024-09-17T23:44:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:44:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 2 2024-09-17T23:44:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd1 2024-09-17T23:44:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary2 2024-09-17T23:44:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary2 2024-09-17T23:44:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:44:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary2 2024-09-17T23:44:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:44:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:44:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 3 2024-09-17T23:44:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd2 2024-09-17T23:44:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary3 2024-09-17T23:44:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary3 2024-09-17T23:44:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:44:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:44:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary3 2024-09-17T23:44:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 4 2024-09-17T23:44:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:44:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd3 2024-09-17T23:44:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary4 2024-09-17T23:44:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary4 2024-09-17T23:44:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:44:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary4 2024-09-17T23:44:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:44:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:44:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 5 2024-09-17T23:44:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd4 2024-09-17T23:44:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:44:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary5 2024-09-17T23:44:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary5 2024-09-17T23:44:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary5 2024-09-17T23:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 6 2024-09-17T23:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd5 2024-09-17T23:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary6 2024-09-17T23:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary6 2024-09-17T23:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary6 2024-09-17T23:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:44:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 55852 2024-09-17T23:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 10 2024-09-17T23:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 9 2024-09-17T23:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd9 2024-09-17T23:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd8 2024-09-17T23:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary10 2024-09-17T23:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary10 2024-09-17T23:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:44:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary10 2024-09-17T23:44:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary9 2024-09-17T23:44:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary9 2024-09-17T23:44:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:44:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary9 2024-09-17T23:44:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 8 2024-09-17T23:44:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd7 2024-09-17T23:44:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary8 2024-09-17T23:44:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary8 2024-09-17T23:44:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:44:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary8 2024-09-17T23:44:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 7 2024-09-17T23:44:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd6 2024-09-17T23:44:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary7 2024-09-17T23:44:53.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary7 2024-09-17T23:44:53.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:44:53.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary7 2024-09-17T23:44:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-17T23:44:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:44:54.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-17T23:44:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-17T23:44:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:44:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:44:54.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-17T23:44:54.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-17T23:44:54.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:44:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:44:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-17T23:44:54.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:44:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-17T23:44:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:44:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-17T23:44:54.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:44:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:44:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-17T23:44:54.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-17T23:44:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:45:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=ab9946d61e92eb0225da3729d7f27e86 2024-09-17T23:45:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-17T23:45:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-17T23:45:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:45:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-17T23:45:24.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=76accaf84e68b0921f521a3de39c6d6d 2024-09-17T23:45:24.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-17T23:45:24.762 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:45:24.760+0000 7f706e974640 -1 reset not still connected to 0x55df20caac30 2024-09-17T23:45:24.763 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:45:24.761+0000 7f706e974640 -1 reset not still connected to 0x55df2db0a9c0 2024-09-17T23:45:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-17T23:45:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:45:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-17T23:45:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=55bba8eccbfea5d6c0112b6fc5299bb4 2024-09-17T23:45:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-17T23:45:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-17T23:45:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:45:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-17T23:45:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=107d978218cd67edd62811bc91b28aa6 2024-09-17T23:45:24.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-17T23:45:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-17T23:45:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:45:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-17T23:45:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=567cc10fc27250ab19f219b271b799b7 2024-09-17T23:45:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-17T23:45:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=11f2f5910d842800de1c4e80dc1a1666 2024-09-17T23:45:25.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-17T23:45:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=42dae60326ec117893ef71fbe672d6f1 2024-09-17T23:45:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-17T23:45:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-17T23:45:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:45:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-17T23:45:25.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-17T23:45:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:45:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-17T23:45:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-17T23:45:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:45:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-17T23:45:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=fe72bea9dfc40dfb9ce4b8c815165c59 2024-09-17T23:45:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-17T23:45:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=dcf187e5c93c31f2efe5d8f1e3f62179 2024-09-17T23:45:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-17T23:45:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-17T23:45:25.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:45:25.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-17T23:45:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-17T23:45:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:45:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-17T23:45:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=5cc7cfed9108683e6df0cf8e0c2dbbee 2024-09-17T23:45:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-17T23:45:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-17T23:45:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:45:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-17T23:45:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:45:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:45:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-17T23:45:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:45:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:45:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:45:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:45:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-17T23:45:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:45:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-17T23:45:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:45:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-17T23:45:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:45:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-17T23:45:26.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:45:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:45:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:45:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-17T23:45:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:45:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-17T23:45:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:45:27.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:45:27.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:45:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-17T23:45:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:45:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-17T23:45:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-17T23:45:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:45:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:27.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-17T23:45:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:45:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:27.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:27.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-17T23:45:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:45:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-17T23:45:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-17T23:45:27.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:45:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:27.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-17T23:45:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-17T23:45:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:45:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-17T23:45:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:45:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:27.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-17T23:45:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:27.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:27.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:45:27.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:27.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:27.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:27.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:27.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-17T23:45:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:45:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-17T23:45:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-17T23:45:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8a8eecbf-6b3c-4a59-8cac-b25382616701 2024-09-17T23:45:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27 2024-09-17T23:45:27.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:45:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27' 2024-09-17T23:45:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:27.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:27.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:45:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-17T23:45:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9409f57e-7aad-46f4-8c9e-6fe60e04c0ae 2024-09-17T23:45:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27 2024-09-17T23:45:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:45:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27' 2024-09-17T23:45:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-17T23:45:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9b073ae2-3e3a-4e6c-af00-5c64a0ab7d7d 2024-09-17T23:45:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27 2024-09-17T23:45:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:45:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27' 2024-09-17T23:45:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:45:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-17 23:45:27' 2024-09-17T23:45:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:45:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:27' 2024-09-17T23:45:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:45:27.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-17T23:45:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: dc1f808f-0bc4-4dc3-928e-846d00114044 2024-09-17T23:45:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27 2024-09-17T23:45:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:45:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27' 2024-09-17T23:45:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-17 23:45:27' 2024-09-17T23:45:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:45:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:27' 2024-09-17T23:45:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:27.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:27.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:45:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-17 23:45:27' 2024-09-17T23:45:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:45:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:27' 2024-09-17T23:45:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-17 23:45:27' 2024-09-17T23:45:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:45:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:27' 2024-09-17T23:45:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-17T23:45:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: fd8e79f3-d28a-4074-9d90-015d42d369a6 2024-09-17T23:45:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27 2024-09-17T23:45:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:45:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27' 2024-09-17T23:45:27.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:27.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:45:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:27.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:27.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-17 23:45:27' 2024-09-17T23:45:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:45:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:27' 2024-09-17T23:45:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2609152.0,"last_snapshot_bytes":3493888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616530,"remote_snapshot_timestamp":1726616530,"replay_state":"idle"}2024-09-17 23:45:17image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1730560.0,"last_snapshot_bytes":1724416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616534,"remote_snapshot_timestamp":1726616534,"replay_state":"idle"}2024-09-17 23:45:17image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1689600.0,"last_snapshot_bytes":1712128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616535,"remote_snapshot_timestamp":1726616535,"replay_state":"idle"}2024-09-17 23:45:17image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1726464.0,"last_snapshot_bytes":1720320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616536,"remote_snapshot_timestamp":1726616536,"replay_state":"idle"}2024-09-17 23:45:17image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1669120.0,"last_snapshot_bytes":1540096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616537,"remote_snapshot_timestamp":1726616537,"replay_state":"idle"}2024-09-17 23:45:17' 2024-09-17T23:45:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-17T23:45:28.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-17T23:45:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-17T23:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2609152.0,"last_snapshot_bytes":3493888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616530,"remote_snapshot_timestamp":1726616530,"replay_state":"idle"}2024-09-17 23:45:17image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1730560.0,"last_snapshot_bytes":1724416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616534,"remote_snapshot_timestamp":1726616534,"replay_state":"idle"}2024-09-17 23:45:17image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1689600.0,"last_snapshot_bytes":1712128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616535,"remote_snapshot_timestamp":1726616535,"replay_state":"idle"}2024-09-17 23:45:17image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1726464.0,"last_snapshot_bytes":1720320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616536,"remote_snapshot_timestamp":1726616536,"replay_state":"idle"}2024-09-17 23:45:17image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1669120.0,"last_snapshot_bytes":1540096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616537,"remote_snapshot_timestamp":1726616537,"replay_state":"idle"}2024-09-17 23:45:17' 2024-09-17T23:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-17T23:45:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2609152.0,"last_snapshot_bytes":3493888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616530,"remote_snapshot_timestamp":1726616530,"replay_state":"idle"}2024-09-17 23:45:17image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1730560.0,"last_snapshot_bytes":1724416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616534,"remote_snapshot_timestamp":1726616534,"replay_state":"idle"}2024-09-17 23:45:17image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1689600.0,"last_snapshot_bytes":1712128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616535,"remote_snapshot_timestamp":1726616535,"replay_state":"idle"}2024-09-17 23:45:17image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1726464.0,"last_snapshot_bytes":1720320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616536,"remote_snapshot_timestamp":1726616536,"replay_state":"idle"}2024-09-17 23:45:17image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1669120.0,"last_snapshot_bytes":1540096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616537,"remote_snapshot_timestamp":1726616537,"replay_state":"idle"}2024-09-17 23:45:17' 2024-09-17T23:45:28.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:28.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-17T23:45:28.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:28.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-17T23:45:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-17T23:45:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:45:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:28.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-17T23:45:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-17T23:45:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:45:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2609152.0,"last_snapshot_bytes":3493888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616530,"remote_snapshot_timestamp":1726616530,"replay_state":"idle"}2024-09-17 23:45:17image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1730560.0,"last_snapshot_bytes":1724416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616534,"remote_snapshot_timestamp":1726616534,"replay_state":"idle"}2024-09-17 23:45:17image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1689600.0,"last_snapshot_bytes":1712128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616535,"remote_snapshot_timestamp":1726616535,"replay_state":"idle"}2024-09-17 23:45:17image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1726464.0,"last_snapshot_bytes":1720320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616536,"remote_snapshot_timestamp":1726616536,"replay_state":"idle"}2024-09-17 23:45:17image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1669120.0,"last_snapshot_bytes":1540096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616537,"remote_snapshot_timestamp":1726616537,"replay_state":"idle"}2024-09-17 23:45:17' 2024-09-17T23:45:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-17T23:45:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-17T23:45:28.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:28.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-17T23:45:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:45:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:28.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2609152.0,"last_snapshot_bytes":3493888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616530,"remote_snapshot_timestamp":1726616530,"replay_state":"idle"}2024-09-17 23:45:17image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1730560.0,"last_snapshot_bytes":1724416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616534,"remote_snapshot_timestamp":1726616534,"replay_state":"idle"}2024-09-17 23:45:17image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1689600.0,"last_snapshot_bytes":1712128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616535,"remote_snapshot_timestamp":1726616535,"replay_state":"idle"}2024-09-17 23:45:17image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1726464.0,"last_snapshot_bytes":1720320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616536,"remote_snapshot_timestamp":1726616536,"replay_state":"idle"}2024-09-17 23:45:17image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1669120.0,"last_snapshot_bytes":1540096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616537,"remote_snapshot_timestamp":1726616537,"replay_state":"idle"}2024-09-17 23:45:17' 2024-09-17T23:45:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-17T23:45:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-17T23:45:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-17T23:45:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:45:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-17T23:45:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:45:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-17T23:45:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-17T23:45:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:45:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-17T23:45:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-17T23:45:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:45:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-17T23:45:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-17T23:45:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 3c802c28-d835-428e-9487-2297aed8b734 2024-09-17T23:45:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27 2024-09-17T23:45:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:45:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:45:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1669120.0,"last_snapshot_bytes":1540096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616537,"remote_snapshot_timestamp":1726616537,"replay_state":"idle"} 2024-09-17T23:45:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:17' 2024-09-17T23:45:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:45:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:28.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-17 23:45:27' 2024-09-17T23:45:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:45:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:27' 2024-09-17T23:45:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:28.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-17T23:45:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 7d1adc6c-e30d-475c-ba6c-ffbffadfed3a 2024-09-17T23:45:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27 2024-09-17T23:45:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:45:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:45:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1726464.0,"last_snapshot_bytes":1720320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616536,"remote_snapshot_timestamp":1726616536,"replay_state":"idle"} 2024-09-17T23:45:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:17' 2024-09-17T23:45:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:28.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:45:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:28.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-17T23:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-17T23:45:28.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-17T23:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-17T23:45:28.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-17 23:45:27' 2024-09-17T23:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:45:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:27' 2024-09-17T23:45:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-17T23:45:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 2683111f-ced2-4294-a3f5-061efd582a2c 2024-09-17T23:45:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27 2024-09-17T23:45:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:45:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:45:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1689600.0,"last_snapshot_bytes":1712128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616535,"remote_snapshot_timestamp":1726616535,"replay_state":"idle"} 2024-09-17T23:45:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:17' 2024-09-17T23:45:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:45:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-17 23:45:27' 2024-09-17T23:45:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:45:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:27' 2024-09-17T23:45:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-17T23:45:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 83c782ac-6390-4046-a744-fa055dd473a9 2024-09-17T23:45:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27 2024-09-17T23:45:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:45:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:45:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1730560.0,"last_snapshot_bytes":1724416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616534,"remote_snapshot_timestamp":1726616534,"replay_state":"idle"} 2024-09-17T23:45:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:17' 2024-09-17T23:45:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:45:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:28.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:28.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-17 23:45:27' 2024-09-17T23:45:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:45:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:27' 2024-09-17T23:45:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-17T23:45:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 5d47f760-049a-45f6-b08b-cc829d2fd297 2024-09-17T23:45:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27 2024-09-17T23:45:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:45:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:45:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2609152.0,"last_snapshot_bytes":3493888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616530,"remote_snapshot_timestamp":1726616530,"replay_state":"idle"} 2024-09-17T23:45:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:17' 2024-09-17T23:45:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:28.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:45:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-17 23:45:27' 2024-09-17T23:45:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:45:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:45:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:27' 2024-09-17T23:45:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2609152.0,"last_snapshot_bytes":3493888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616530,"remote_snapshot_timestamp":1726616530,"replay_state":"idle"}2024-09-17 23:45:17image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1730560.0,"last_snapshot_bytes":1724416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616534,"remote_snapshot_timestamp":1726616534,"replay_state":"idle"}2024-09-17 23:45:17image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1689600.0,"last_snapshot_bytes":1712128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616535,"remote_snapshot_timestamp":1726616535,"replay_state":"idle"}2024-09-17 23:45:17image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1726464.0,"last_snapshot_bytes":1720320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616536,"remote_snapshot_timestamp":1726616536,"replay_state":"idle"}2024-09-17 23:45:17image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1669120.0,"last_snapshot_bytes":1540096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616537,"remote_snapshot_timestamp":1726616537,"replay_state":"idle"}2024-09-17 23:45:17' 2024-09-17T23:45:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-17T23:45:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:28.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-17T23:45:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2609152.0,"last_snapshot_bytes":3493888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616530,"remote_snapshot_timestamp":1726616530,"replay_state":"idle"}2024-09-17 23:45:17image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1730560.0,"last_snapshot_bytes":1724416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616534,"remote_snapshot_timestamp":1726616534,"replay_state":"idle"}2024-09-17 23:45:17image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1689600.0,"last_snapshot_bytes":1712128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616535,"remote_snapshot_timestamp":1726616535,"replay_state":"idle"}2024-09-17 23:45:17image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1726464.0,"last_snapshot_bytes":1720320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616536,"remote_snapshot_timestamp":1726616536,"replay_state":"idle"}2024-09-17 23:45:17image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1669120.0,"last_snapshot_bytes":1540096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616537,"remote_snapshot_timestamp":1726616537,"replay_state":"idle"}2024-09-17 23:45:17' 2024-09-17T23:45:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-17T23:45:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:45:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:28.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-17T23:45:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-17T23:45:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-17T23:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2609152.0,"last_snapshot_bytes":3493888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616530,"remote_snapshot_timestamp":1726616530,"replay_state":"idle"}2024-09-17 23:45:17image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1730560.0,"last_snapshot_bytes":1724416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616534,"remote_snapshot_timestamp":1726616534,"replay_state":"idle"}2024-09-17 23:45:17image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1689600.0,"last_snapshot_bytes":1712128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616535,"remote_snapshot_timestamp":1726616535,"replay_state":"idle"}2024-09-17 23:45:17image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1726464.0,"last_snapshot_bytes":1720320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616536,"remote_snapshot_timestamp":1726616536,"replay_state":"idle"}2024-09-17 23:45:17image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1669120.0,"last_snapshot_bytes":1540096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616537,"remote_snapshot_timestamp":1726616537,"replay_state":"idle"}2024-09-17 23:45:17' 2024-09-17T23:45:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-17T23:45:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-17T23:45:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-17T23:45:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:45:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2609152.0,"last_snapshot_bytes":3493888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616530,"remote_snapshot_timestamp":1726616530,"replay_state":"idle"}2024-09-17 23:45:17image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1730560.0,"last_snapshot_bytes":1724416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616534,"remote_snapshot_timestamp":1726616534,"replay_state":"idle"}2024-09-17 23:45:17image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1689600.0,"last_snapshot_bytes":1712128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616535,"remote_snapshot_timestamp":1726616535,"replay_state":"idle"}2024-09-17 23:45:17image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1726464.0,"last_snapshot_bytes":1720320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616536,"remote_snapshot_timestamp":1726616536,"replay_state":"idle"}2024-09-17 23:45:17image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1669120.0,"last_snapshot_bytes":1540096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616537,"remote_snapshot_timestamp":1726616537,"replay_state":"idle"}2024-09-17 23:45:17' 2024-09-17T23:45:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-17T23:45:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-17T23:45:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-17T23:45:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:45:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2609152.0,"last_snapshot_bytes":3493888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616530,"remote_snapshot_timestamp":1726616530,"replay_state":"idle"}2024-09-17 23:45:17image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1730560.0,"last_snapshot_bytes":1724416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616534,"remote_snapshot_timestamp":1726616534,"replay_state":"idle"}2024-09-17 23:45:17image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1689600.0,"last_snapshot_bytes":1712128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616535,"remote_snapshot_timestamp":1726616535,"replay_state":"idle"}2024-09-17 23:45:17image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1726464.0,"last_snapshot_bytes":1720320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616536,"remote_snapshot_timestamp":1726616536,"replay_state":"idle"}2024-09-17 23:45:17image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:45:27cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1669120.0,"last_snapshot_bytes":1540096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616537,"remote_snapshot_timestamp":1726616537,"replay_state":"idle"}2024-09-17 23:45:17' 2024-09-17T23:45:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-17T23:45:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:28.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-17T23:45:28.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:28.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-17T23:45:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:45:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:45:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-17T23:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-17T23:45:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-17T23:45:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:45:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-17T23:45:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-17T23:45:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:45:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-17T23:45:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-17T23:45:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:45:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-17T23:45:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-17T23:45:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:45:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-17T23:45:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-17T23:45:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9b073ae2-3e3a-4e6c-af00-5c64a0ab7d7d 2024-09-17T23:45:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27 2024-09-17T23:45:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:45:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27' 2024-09-17T23:45:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:29.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-17T23:45:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8a8eecbf-6b3c-4a59-8cac-b25382616701 2024-09-17T23:45:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27 2024-09-17T23:45:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:45:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27' 2024-09-17T23:45:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:45:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:45:29.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:29.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-17 23:45:27' 2024-09-17T23:45:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:45:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:27' 2024-09-17T23:45:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-17 23:45:27' 2024-09-17T23:45:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:45:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:27' 2024-09-17T23:45:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-17T23:45:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: dc1f808f-0bc4-4dc3-928e-846d00114044 2024-09-17T23:45:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27 2024-09-17T23:45:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:45:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27' 2024-09-17T23:45:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:45:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-17 23:45:27' 2024-09-17T23:45:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:45:29.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:27' 2024-09-17T23:45:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-17T23:45:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: fd8e79f3-d28a-4074-9d90-015d42d369a6 2024-09-17T23:45:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27 2024-09-17T23:45:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:45:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27' 2024-09-17T23:45:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:45:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:29.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:29.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:29.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-17 23:45:27' 2024-09-17T23:45:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:45:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:27' 2024-09-17T23:45:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27' 2024-09-17T23:45:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-17T23:45:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:29.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-17T23:45:29.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27' 2024-09-17T23:45:29.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-17T23:45:29.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-17T23:45:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:45:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:45:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-17T23:45:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-17T23:45:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-17T23:45:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:45:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:45:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-17T23:45:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-17T23:45:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9409f57e-7aad-46f4-8c9e-6fe60e04c0ae 2024-09-17T23:45:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27 2024-09-17T23:45:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:45:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27' 2024-09-17T23:45:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:45:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:29.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27' 2024-09-17T23:45:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-17T23:45:29.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:29.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:29.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:29.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-17T23:45:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-17 23:45:27' 2024-09-17T23:45:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:45:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:27' 2024-09-17T23:45:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-17T23:45:29.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:45:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:45:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-17T23:45:29.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27' 2024-09-17T23:45:29.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:29.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-17T23:45:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-17T23:45:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:29.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-17T23:45:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:45:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:45:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-17T23:45:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27' 2024-09-17T23:45:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-17T23:45:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:27' 2024-09-17T23:45:29.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-17T23:45:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-17T23:45:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:45:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:45:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-17T23:45:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-17T23:45:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:45:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-17T23:45:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-17T23:45:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:45:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:29.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-17T23:45:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-17T23:45:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:45:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-17T23:45:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-17T23:45:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 3c802c28-d835-428e-9487-2297aed8b734 2024-09-17T23:45:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:45:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:29 2024-09-17T23:45:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:45:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27' 2024-09-17T23:45:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:29.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:45:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:29' 2024-09-17T23:45:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-17 23:45:29' 2024-09-17T23:45:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:45:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:29' 2024-09-17T23:45:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-17T23:45:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 7d1adc6c-e30d-475c-ba6c-ffbffadfed3a 2024-09-17T23:45:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:45:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:29 2024-09-17T23:45:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:45:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27' 2024-09-17T23:45:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:45:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:29' 2024-09-17T23:45:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-17 23:45:29' 2024-09-17T23:45:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:45:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:29' 2024-09-17T23:45:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-17T23:45:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:45:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-17T23:45:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-17T23:45:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:45:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:45:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:45:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-17T23:45:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-17T23:45:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 2683111f-ced2-4294-a3f5-061efd582a2c 2024-09-17T23:45:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:45:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:29 2024-09-17T23:45:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:45:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27' 2024-09-17T23:45:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:45:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:29' 2024-09-17T23:45:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-17 23:45:29' 2024-09-17T23:45:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:45:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:29' 2024-09-17T23:45:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-17T23:45:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 83c782ac-6390-4046-a744-fa055dd473a9 2024-09-17T23:45:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:45:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:29 2024-09-17T23:45:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:45:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27' 2024-09-17T23:45:29.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:45:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:29' 2024-09-17T23:45:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-17 23:45:29' 2024-09-17T23:45:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:45:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:29' 2024-09-17T23:45:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-17T23:45:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 5d47f760-049a-45f6-b08b-cc829d2fd297 2024-09-17T23:45:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:45:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:45:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:29 2024-09-17T23:45:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:45:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:45:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:45:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:45:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:27' 2024-09-17T23:45:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:45:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:45:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:45:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:45:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:45:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:45:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27' 2024-09-17T23:45:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:45:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-17T23:45:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:45:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:45:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27' 2024-09-17T23:45:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:45:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-17T23:45:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:29' 2024-09-17T23:45:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-17T23:45:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:29' 2024-09-17T23:45:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-17 23:45:29' 2024-09-17T23:45:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:45:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:45:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:29' 2024-09-17T23:45:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:45:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:45:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:45:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:29' 2024-09-17T23:45:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-17T23:45:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-17T23:45:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:45:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:45:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-17T23:45:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-17T23:45:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:45:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:45:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-17T23:45:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27' 2024-09-17T23:45:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:30.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-17T23:45:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:29' 2024-09-17T23:45:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-17T23:45:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-17T23:45:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:45:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:45:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-17T23:45:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27' 2024-09-17T23:45:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-17T23:45:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:29' 2024-09-17T23:45:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-17T23:45:30.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-17T23:45:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:45:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:45:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-17T23:45:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:45:27cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary683c782ac-6390-4046-a744-fa055dd473a9up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27image-primary93c802c28-d835-428e-9487-2297aed8b734up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:45:29cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:45:27' 2024-09-17T23:45:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:45:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-17T23:45:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:29' 2024-09-17T23:45:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-17T23:45:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:45:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:45:30.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:45:30.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:45:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:45:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-17T23:45:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:45:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:45:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:45:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:45:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-17T23:45:30.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:45:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:45:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:45:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:45:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:45:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:45:30.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:45:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:45:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:45:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-17T23:45:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:45:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:45:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:45:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:45:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-17T23:45:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-17T23:45:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:45:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:45:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-17T23:45:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:45:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:45:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:45:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:45:30.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-17T23:45:30.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-17T23:45:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:45:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:45:30.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-17T23:45:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd0 2024-09-17T23:45:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-17T23:45:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:45:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd4 2024-09-17T23:45:30.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-17T23:45:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:45:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd1 2024-09-17T23:45:30.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-17T23:45:30.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:45:30.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd3 2024-09-17T23:45:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:45:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-17T23:45:30.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd2 2024-09-17T23:45:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-17T23:45:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd6 2024-09-17T23:45:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:45:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-17T23:45:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:45:30.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd5 2024-09-17T23:45:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-17T23:45:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:45:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:45:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:45:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:45:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:45:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:45:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:45:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:45:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:45:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:45:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-17T23:45:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-17T23:45:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-17T23:45:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd7 2024-09-17T23:45:31.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-17T23:45:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:45:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd8 2024-09-17T23:45:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-17T23:45:31.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd9 2024-09-17T23:45:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:45:31.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-17T23:45:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:46:00.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=567cc10fc27250ab19f219b271b799b7 2024-09-17T23:46:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-17T23:46:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=5cc7cfed9108683e6df0cf8e0c2dbbee 2024-09-17T23:46:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-17T23:46:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 567cc10fc27250ab19f219b271b799b7 != \5\6\7\c\c\1\0\f\c\2\7\2\5\0\a\b\1\9\f\2\1\9\b\2\7\1\b\7\9\9\b\7 ]] 2024-09-17T23:46:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 5cc7cfed9108683e6df0cf8e0c2dbbee != \5\c\c\7\c\f\e\d\9\1\0\8\6\8\3\e\6\d\f\0\c\f\8\e\0\c\2\d\b\b\e\e ]] 2024-09-17T23:46:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=fe72bea9dfc40dfb9ce4b8c815165c59 2024-09-17T23:46:01.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-17T23:46:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ fe72bea9dfc40dfb9ce4b8c815165c59 != \f\e\7\2\b\e\a\9\d\f\c\4\0\d\f\b\9\c\e\4\b\8\c\8\1\5\1\6\5\c\5\9 ]] 2024-09-17T23:46:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=107d978218cd67edd62811bc91b28aa6 2024-09-17T23:46:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-17T23:46:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 107d978218cd67edd62811bc91b28aa6 != \1\0\7\d\9\7\8\2\1\8\c\d\6\7\e\d\d\6\2\8\1\1\b\c\9\1\b\2\8\a\a\6 ]] 2024-09-17T23:46:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:46:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 55853 2024-09-17T23:46:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=76accaf84e68b0921f521a3de39c6d6d 2024-09-17T23:46:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-17T23:46:02.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 76accaf84e68b0921f521a3de39c6d6d != \7\6\a\c\c\a\f\8\4\e\6\8\b\0\9\2\1\f\5\2\1\a\3\d\e\3\9\c\6\d\6\d ]] 2024-09-17T23:46:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=55bba8eccbfea5d6c0112b6fc5299bb4 2024-09-17T23:46:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-17T23:46:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=ab9946d61e92eb0225da3729d7f27e86 2024-09-17T23:46:02.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-17T23:46:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 55bba8eccbfea5d6c0112b6fc5299bb4 != \5\5\b\b\a\8\e\c\c\b\f\e\a\5\d\6\c\0\1\1\2\b\6\f\c\5\2\9\9\b\b\4 ]] 2024-09-17T23:46:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ ab9946d61e92eb0225da3729d7f27e86 != \a\b\9\9\4\6\d\6\1\e\9\2\e\b\0\2\2\5\d\a\3\7\2\9\d\7\f\2\7\e\8\6 ]] 2024-09-17T23:46:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=42dae60326ec117893ef71fbe672d6f1 2024-09-17T23:46:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-17T23:46:02.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 42dae60326ec117893ef71fbe672d6f1 != \4\2\d\a\e\6\0\3\2\6\e\c\1\1\7\8\9\3\e\f\7\1\f\b\e\6\7\2\d\6\f\1 ]] 2024-09-17T23:46:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=11f2f5910d842800de1c4e80dc1a1666 2024-09-17T23:46:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-17T23:46:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 11f2f5910d842800de1c4e80dc1a1666 != \1\1\f\2\f\5\9\1\0\d\8\4\2\8\0\0\d\e\1\c\4\e\8\0\d\c\1\a\1\6\6\6 ]] 2024-09-17T23:46:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:46:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 55855 2024-09-17T23:46:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:46:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 55857 2024-09-17T23:46:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:46:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 55859 2024-09-17T23:46:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:46:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 55861 2024-09-17T23:46:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:46:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 55863 2024-09-17T23:46:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=dcf187e5c93c31f2efe5d8f1e3f62179 2024-09-17T23:46:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-17T23:46:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ dcf187e5c93c31f2efe5d8f1e3f62179 != \d\c\f\1\8\7\e\5\c\9\3\c\3\1\f\2\e\f\e\5\d\8\f\1\e\3\f\6\2\1\7\9 ]] 2024-09-17T23:46:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:46:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 55865 2024-09-17T23:46:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:46:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 55866 2024-09-17T23:46:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:46:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 55867 2024-09-17T23:46:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-17T23:46:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-17T23:46:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-17T23:46:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:46:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-17T23:46:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:03.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:03.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:46:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-17T23:46:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-17T23:46:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-17T23:46:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8a8eecbf-6b3c-4a59-8cac-b25382616701 2024-09-17T23:46:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:46:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":6424166.4,"bytes_per_snapshot":64241664.0,"last_snapshot_bytes":64241664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"} 2024-09-17T23:46:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:46:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:47 2024-09-17T23:46:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:46:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:46:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:46:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:46:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:46:02' 2024-09-17T23:46:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:46:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:46:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:46:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":6424166.4,"bytes_per_snapshot":64241664.0,"last_snapshot_bytes":64241664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"} 2024-09-17T23:46:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:46:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:46:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:46:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:46:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:46:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:46:04.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:47' 2024-09-17T23:46:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-17 23:45:47' 2024-09-17T23:46:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:47' 2024-09-17T23:46:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:46:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18a8eecbf-6b3c-4a59-8cac-b25382616701up+replayingreplaying, {"bytes_per_second":6424166.4,"bytes_per_snapshot":64241664.0,"last_snapshot_bytes":64241664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:02image-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+replayingreplaying, {"bytes_per_second":6791577.6,"bytes_per_snapshot":67915776.0,"last_snapshot_bytes":67915776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:02image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+replayingreplaying, {"bytes_per_second":6379929.6,"bytes_per_snapshot":63799296.0,"last_snapshot_bytes":63799296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:02image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+replayingreplaying, {"bytes_per_second":6349209.6,"bytes_per_snapshot":63492096.0,"last_snapshot_bytes":63492096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:02image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+replayingreplaying, {"bytes_per_second":7172096.0,"bytes_per_snapshot":71720960.0,"last_snapshot_bytes":71720960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:02image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+replayingreplaying, {"bytes_per_second":7228211.2,"bytes_per_snapshot":72282112.0,"last_snapshot_bytes":72282112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:02image-primary683c782ac-6390-4046-a744-fa055dd473a9up+replayingreplaying, {"bytes_per_second":7184384.0,"bytes_per_snapshot":71843840.0,"last_snapshot_bytes":71843840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:02image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+replayingreplaying, {"bytes_per_second":7145472.0,"bytes_per_snapshot":71454720.0,"last_snapshot_bytes":71454720,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:02image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+replayingreplaying, {"bytes_per_second":7159398.4,"bytes_per_snapshot":71593984.0,"last_snapshot_bytes":71593984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:02image-primary93c802c28-d835-428e-9487-2297aed8b734up+replayingreplaying, {"bytes_per_second":7241318.4,"bytes_per_snapshot":72413184.0,"last_snapshot_bytes":72413184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:02' 2024-09-17T23:46:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:46:04.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-17T23:46:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:47' 2024-09-17T23:46:04.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-17T23:46:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:46:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:46:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-17T23:46:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:46:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:04.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:04.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-17T23:46:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-17T23:46:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:46:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-17T23:46:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:46:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary1 2024-09-17T23:46:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:06.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary1 2024-09-17T23:46:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-17T23:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-17T23:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-17T23:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:46:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-17T23:46:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-17T23:46:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-17T23:46:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 5d47f760-049a-45f6-b08b-cc829d2fd297 2024-09-17T23:46:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:46:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":6379929.6,"bytes_per_snapshot":63799296.0,"last_snapshot_bytes":63799296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"} 2024-09-17T23:46:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:46:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:47 2024-09-17T23:46:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:46:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:46:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:46:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:46:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:46:07' 2024-09-17T23:46:07.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:46:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:46:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:46:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":6379929.6,"bytes_per_snapshot":63799296.0,"last_snapshot_bytes":63799296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"} 2024-09-17T23:46:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:46:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:46:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:46:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:46:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:46:07.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:46:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:47' 2024-09-17T23:46:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-17 23:45:47' 2024-09-17T23:46:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:07.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:47' 2024-09-17T23:46:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:07.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:46:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK942634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+replayingreplaying, {"bytes_per_second":6791577.6,"bytes_per_snapshot":67915776.0,"last_snapshot_bytes":67915776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07image-primary25d47f760-049a-45f6-b08b-cc829d2fd297up+replayingreplaying, {"bytes_per_second":6379929.6,"bytes_per_snapshot":63799296.0,"last_snapshot_bytes":63799296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+replayingreplaying, {"bytes_per_second":6349209.6,"bytes_per_snapshot":63492096.0,"last_snapshot_bytes":63492096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+replayingreplaying, {"bytes_per_second":7172096.0,"bytes_per_snapshot":71720960.0,"last_snapshot_bytes":71720960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+replayingreplaying, {"bytes_per_second":7228211.2,"bytes_per_snapshot":72282112.0,"last_snapshot_bytes":72282112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07image-primary683c782ac-6390-4046-a744-fa055dd473a9up+replayingreplaying, {"bytes_per_second":7184384.0,"bytes_per_snapshot":71843840.0,"last_snapshot_bytes":71843840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+replayingreplaying, {"bytes_per_second":7145472.0,"bytes_per_snapshot":71454720.0,"last_snapshot_bytes":71454720,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+replayingreplaying, {"bytes_per_second":7159398.4,"bytes_per_snapshot":71593984.0,"last_snapshot_bytes":71593984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07image-primary93c802c28-d835-428e-9487-2297aed8b734up+replayingreplaying, {"bytes_per_second":7241318.4,"bytes_per_snapshot":72413184.0,"last_snapshot_bytes":72413184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07' 2024-09-17T23:46:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:46:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-17T23:46:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:47' 2024-09-17T23:46:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-17T23:46:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:46:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:46:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-17T23:46:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:46:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-17T23:46:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-17T23:46:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:46:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-17T23:46:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:46:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary2 2024-09-17T23:46:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary2 2024-09-17T23:46:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-17T23:46:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-17T23:46:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:46:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:46:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-17T23:46:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:46:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:46:10.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-17T23:46:10.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:10.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:10.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:46:10.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:10.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:10.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:10.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:10.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:10.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-17T23:46:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-17T23:46:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: dc1f808f-0bc4-4dc3-928e-846d00114044 2024-09-17T23:46:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:46:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":6349209.6,"bytes_per_snapshot":63492096.0,"last_snapshot_bytes":63492096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"} 2024-09-17T23:46:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:46:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:47 2024-09-17T23:46:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:46:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:46:10.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:46:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:46:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:46:07' 2024-09-17T23:46:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:46:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:46:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:46:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":6349209.6,"bytes_per_snapshot":63492096.0,"last_snapshot_bytes":63492096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"} 2024-09-17T23:46:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:46:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:10.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:46:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:46:10.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:46:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:46:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:46:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:47' 2024-09-17T23:46:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-17 23:45:47' 2024-09-17T23:46:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:46:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:47' 2024-09-17T23:46:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:10.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:46:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK842634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+replayingreplaying, {"bytes_per_second":6791577.6,"bytes_per_snapshot":67915776.0,"last_snapshot_bytes":67915776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07image-primary3dc1f808f-0bc4-4dc3-928e-846d00114044up+replayingreplaying, {"bytes_per_second":6349209.6,"bytes_per_snapshot":63492096.0,"last_snapshot_bytes":63492096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+replayingreplaying, {"bytes_per_second":7172096.0,"bytes_per_snapshot":71720960.0,"last_snapshot_bytes":71720960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+replayingreplaying, {"bytes_per_second":7228211.2,"bytes_per_snapshot":72282112.0,"last_snapshot_bytes":72282112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07image-primary683c782ac-6390-4046-a744-fa055dd473a9up+replayingreplaying, {"bytes_per_second":7184384.0,"bytes_per_snapshot":71843840.0,"last_snapshot_bytes":71843840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+replayingreplaying, {"bytes_per_second":7145472.0,"bytes_per_snapshot":71454720.0,"last_snapshot_bytes":71454720,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+replayingreplaying, {"bytes_per_second":7159398.4,"bytes_per_snapshot":71593984.0,"last_snapshot_bytes":71593984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07image-primary93c802c28-d835-428e-9487-2297aed8b734up+replayingreplaying, {"bytes_per_second":7241318.4,"bytes_per_snapshot":72413184.0,"last_snapshot_bytes":72413184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:07' 2024-09-17T23:46:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:46:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-17T23:46:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:47' 2024-09-17T23:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-17T23:46:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:46:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:46:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-17T23:46:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-17T23:46:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-17T23:46:12.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:46:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-17T23:46:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:46:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:46:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary3 2024-09-17T23:46:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary3 2024-09-17T23:46:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-17T23:46:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-17T23:46:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:46:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:46:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-17T23:46:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:46:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:46:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-17T23:46:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:46:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-17T23:46:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-17T23:46:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9409f57e-7aad-46f4-8c9e-6fe60e04c0ae 2024-09-17T23:46:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:46:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":7172096.0,"bytes_per_snapshot":71720960.0,"last_snapshot_bytes":71720960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"} 2024-09-17T23:46:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:46:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:47 2024-09-17T23:46:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:46:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:46:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:46:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:46:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:46:12' 2024-09-17T23:46:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:46:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:46:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:46:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":7172096.0,"bytes_per_snapshot":71720960.0,"last_snapshot_bytes":71720960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"} 2024-09-17T23:46:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:46:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:46:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:46:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:46:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:46:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:46:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:47' 2024-09-17T23:46:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-17 23:45:47' 2024-09-17T23:46:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:46:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:47' 2024-09-17T23:46:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:46:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK742634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+replayingreplaying, {"bytes_per_second":6791577.6,"bytes_per_snapshot":67915776.0,"last_snapshot_bytes":67915776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:12image-primary49409f57e-7aad-46f4-8c9e-6fe60e04c0aeup+replayingreplaying, {"bytes_per_second":7172096.0,"bytes_per_snapshot":71720960.0,"last_snapshot_bytes":71720960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:12image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+replayingreplaying, {"bytes_per_second":7228211.2,"bytes_per_snapshot":72282112.0,"last_snapshot_bytes":72282112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:12image-primary683c782ac-6390-4046-a744-fa055dd473a9up+replayingreplaying, {"bytes_per_second":7184384.0,"bytes_per_snapshot":71843840.0,"last_snapshot_bytes":71843840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:12image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+replayingreplaying, {"bytes_per_second":7145472.0,"bytes_per_snapshot":71454720.0,"last_snapshot_bytes":71454720,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:12image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+replayingreplaying, {"bytes_per_second":7159398.4,"bytes_per_snapshot":71593984.0,"last_snapshot_bytes":71593984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:12image-primary93c802c28-d835-428e-9487-2297aed8b734up+replayingreplaying, {"bytes_per_second":7241318.4,"bytes_per_snapshot":72413184.0,"last_snapshot_bytes":72413184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:12' 2024-09-17T23:46:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:46:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-17T23:46:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:47' 2024-09-17T23:46:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-17T23:46:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:46:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:13.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:46:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-17T23:46:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:46:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:46:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-17T23:46:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-17T23:46:15.624 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:46:15.621+0000 7f9bfab49640 -1 reset not still connected to 0x55cd07747ba0 2024-09-17T23:46:15.624 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:46:15.621+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782eea0 2024-09-17T23:46:15.624 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:46:15.621+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-17T23:46:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:46:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-17T23:46:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:46:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:46:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary4 2024-09-17T23:46:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary4 2024-09-17T23:46:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-17T23:46:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-17T23:46:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:46:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:46:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-17T23:46:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:46:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:46:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-17T23:46:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:46:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:16.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-17T23:46:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-17T23:46:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: fd8e79f3-d28a-4074-9d90-015d42d369a6 2024-09-17T23:46:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:46:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":7228211.2,"bytes_per_snapshot":72282112.0,"last_snapshot_bytes":72282112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"} 2024-09-17T23:46:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:46:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:45:47 2024-09-17T23:46:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:46:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:46:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:46:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:46:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:46:12' 2024-09-17T23:46:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:46:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:46:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:46:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":7228211.2,"bytes_per_snapshot":72282112.0,"last_snapshot_bytes":72282112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"} 2024-09-17T23:46:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:46:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:46:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:46:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:46:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:46:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:46:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:47' 2024-09-17T23:46:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-17 23:45:47' 2024-09-17T23:46:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:46:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:45:47' 2024-09-17T23:46:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:46:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK642634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+replayingreplaying, {"bytes_per_second":6791577.6,"bytes_per_snapshot":67915776.0,"last_snapshot_bytes":67915776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:12image-primary5fd8e79f3-d28a-4074-9d90-015d42d369a6up+replayingreplaying, {"bytes_per_second":7228211.2,"bytes_per_snapshot":72282112.0,"last_snapshot_bytes":72282112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:12image-primary683c782ac-6390-4046-a744-fa055dd473a9up+replayingreplaying, {"bytes_per_second":7184384.0,"bytes_per_snapshot":71843840.0,"last_snapshot_bytes":71843840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:12image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+replayingreplaying, {"bytes_per_second":7145472.0,"bytes_per_snapshot":71454720.0,"last_snapshot_bytes":71454720,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:12image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+replayingreplaying, {"bytes_per_second":7159398.4,"bytes_per_snapshot":71593984.0,"last_snapshot_bytes":71593984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:12image-primary93c802c28-d835-428e-9487-2297aed8b734up+replayingreplaying, {"bytes_per_second":7241318.4,"bytes_per_snapshot":72413184.0,"last_snapshot_bytes":72413184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:45:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:12' 2024-09-17T23:46:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:46:17.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-17T23:46:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:45:47' 2024-09-17T23:46:17.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-17T23:46:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:46:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:46:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-17T23:46:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:46:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:46:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-17T23:46:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-17T23:46:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:46:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-17T23:46:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:46:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:46:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary5 2024-09-17T23:46:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary5 2024-09-17T23:46:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-17T23:46:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-17T23:46:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:46:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:46:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-17T23:46:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:46:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:46:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-17T23:46:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:46:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-17T23:46:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-17T23:46:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 83c782ac-6390-4046-a744-fa055dd473a9 2024-09-17T23:46:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:46:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71843840.0,"last_snapshot_bytes":71843840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"} 2024-09-17T23:46:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:46:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:46:17 2024-09-17T23:46:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:46:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:46:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:46:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:46:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:46:17' 2024-09-17T23:46:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:46:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:46:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:46:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71843840.0,"last_snapshot_bytes":71843840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"} 2024-09-17T23:46:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:46:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:46:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:46:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:46:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:46:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:46:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:46:17' 2024-09-17T23:46:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-17 23:46:17' 2024-09-17T23:46:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:46:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:46:17' 2024-09-17T23:46:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:46:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK542634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67915776.0,"last_snapshot_bytes":67915776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:46:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:17image-primary683c782ac-6390-4046-a744-fa055dd473a9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71843840.0,"last_snapshot_bytes":71843840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:46:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:17image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71454720.0,"last_snapshot_bytes":71454720,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:46:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:17image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71593984.0,"last_snapshot_bytes":71593984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:46:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:17image-primary93c802c28-d835-428e-9487-2297aed8b734up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72413184.0,"last_snapshot_bytes":72413184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:46:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:17' 2024-09-17T23:46:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:46:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-17T23:46:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:46:17' 2024-09-17T23:46:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-17T23:46:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:46:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:46:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-17T23:46:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:46:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:46:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-17T23:46:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-17T23:46:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:46:21.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-17T23:46:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:46:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:46:21.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary6 2024-09-17T23:46:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary6 2024-09-17T23:46:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-17T23:46:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-17T23:46:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:46:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:46:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-17T23:46:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:46:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:46:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-17T23:46:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:46:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-17T23:46:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-17T23:46:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 2683111f-ced2-4294-a3f5-061efd582a2c 2024-09-17T23:46:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:46:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71454720.0,"last_snapshot_bytes":71454720,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"} 2024-09-17T23:46:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:46:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:46:17 2024-09-17T23:46:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:46:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:46:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:46:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:46:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:46:22' 2024-09-17T23:46:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:46:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:46:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:46:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71454720.0,"last_snapshot_bytes":71454720,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"} 2024-09-17T23:46:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:46:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:46:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:46:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:46:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:46:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:46:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:46:17' 2024-09-17T23:46:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-17 23:46:17' 2024-09-17T23:46:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:46:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:46:17' 2024-09-17T23:46:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:46:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67915776.0,"last_snapshot_bytes":67915776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:46:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:22image-primary72683111f-ced2-4294-a3f5-061efd582a2cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71454720.0,"last_snapshot_bytes":71454720,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616731,"remote_snapshot_timestamp":1726616731,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:46:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:22image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71593984.0,"last_snapshot_bytes":71593984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:46:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:22image-primary93c802c28-d835-428e-9487-2297aed8b734up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72413184.0,"last_snapshot_bytes":72413184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:46:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:22' 2024-09-17T23:46:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:46:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-17T23:46:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:46:17' 2024-09-17T23:46:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-17T23:46:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:46:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:46:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-17T23:46:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:46:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:46:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-17T23:46:22.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-17T23:46:24.290 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:46:24.285+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-17T23:46:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:46:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-17T23:46:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:46:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:46:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary7 2024-09-17T23:46:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary7 2024-09-17T23:46:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-17T23:46:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-17T23:46:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:46:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:46:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-17T23:46:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:46:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:46:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-17T23:46:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:46:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-17T23:46:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-17T23:46:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 7d1adc6c-e30d-475c-ba6c-ffbffadfed3a 2024-09-17T23:46:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:46:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71593984.0,"last_snapshot_bytes":71593984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"} 2024-09-17T23:46:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:46:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:46:17 2024-09-17T23:46:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:46:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:46:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:46:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:46:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:46:22' 2024-09-17T23:46:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:46:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:46:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:46:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71593984.0,"last_snapshot_bytes":71593984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"} 2024-09-17T23:46:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:46:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:46:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:46:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:46:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:46:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:46:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:46:17' 2024-09-17T23:46:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-17 23:46:17' 2024-09-17T23:46:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:46:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:46:17' 2024-09-17T23:46:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:46:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK342634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67915776.0,"last_snapshot_bytes":67915776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:46:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:22image-primary87d1adc6c-e30d-475c-ba6c-ffbffadfed3aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71593984.0,"last_snapshot_bytes":71593984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:46:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:22image-primary93c802c28-d835-428e-9487-2297aed8b734up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72413184.0,"last_snapshot_bytes":72413184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:46:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:22' 2024-09-17T23:46:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:46:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-17T23:46:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:46:17' 2024-09-17T23:46:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-17T23:46:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:46:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:46:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-17T23:46:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:46:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:46:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-17T23:46:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-17T23:46:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:46:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-17T23:46:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:46:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:46:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary8 2024-09-17T23:46:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary8 2024-09-17T23:46:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-17T23:46:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-17T23:46:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:46:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:46:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-17T23:46:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:46:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:46:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-17T23:46:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:46:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-17T23:46:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-17T23:46:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 3c802c28-d835-428e-9487-2297aed8b734 2024-09-17T23:46:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:46:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72413184.0,"last_snapshot_bytes":72413184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"} 2024-09-17T23:46:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:46:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:46:17 2024-09-17T23:46:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:46:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:46:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:46:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:46:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:46:27' 2024-09-17T23:46:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:46:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:46:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:46:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72413184.0,"last_snapshot_bytes":72413184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"} 2024-09-17T23:46:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:46:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:46:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:46:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:46:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:46:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:46:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:46:17' 2024-09-17T23:46:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-17 23:46:17' 2024-09-17T23:46:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:46:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:46:17' 2024-09-17T23:46:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:46:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK242634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67915776.0,"last_snapshot_bytes":67915776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:46:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:27image-primary93c802c28-d835-428e-9487-2297aed8b734up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72413184.0,"last_snapshot_bytes":72413184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:46:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:27' 2024-09-17T23:46:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:46:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-17T23:46:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:46:17' 2024-09-17T23:46:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-17T23:46:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:46:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:46:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-17T23:46:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:46:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:46:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-17T23:46:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-17T23:46:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:46:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-17T23:46:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:46:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:46:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary9 2024-09-17T23:46:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:30.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary9 2024-09-17T23:46:30.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-17T23:46:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-17T23:46:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:46:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:46:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-17T23:46:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:46:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:30.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:30.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:46:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-17T23:46:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:31.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-17T23:46:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-17T23:46:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9b073ae2-3e3a-4e6c-af00-5c64a0ab7d7d 2024-09-17T23:46:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:46:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67915776.0,"last_snapshot_bytes":67915776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"} 2024-09-17T23:46:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:46:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:46:17 2024-09-17T23:46:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:46:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:46:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:46:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:46:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:46:27' 2024-09-17T23:46:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:46:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:46:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:46:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67915776.0,"last_snapshot_bytes":67915776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"} 2024-09-17T23:46:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:46:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:46:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:31.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:46:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:46:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:46:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:46:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:46:17' 2024-09-17T23:46:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-17 23:46:17' 2024-09-17T23:46:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:46:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:46:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:46:17' 2024-09-17T23:46:31.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:46:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:46:31.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:46:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary109b073ae2-3e3a-4e6c-af00-5c64a0ab7d7dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":67915776.0,"last_snapshot_bytes":67915776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616730,"remote_snapshot_timestamp":1726616730,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:46:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:46:27' 2024-09-17T23:46:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:46:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-17T23:46:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:46:17' 2024-09-17T23:46:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-17T23:46:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:46:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:46:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:46:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:46:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-17T23:46:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:46:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:46:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-17T23:46:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-17T23:46:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:46:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-17T23:46:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:46:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:46:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:46:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary10 2024-09-17T23:46:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary10 2024-09-17T23:46:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..10} 2024-09-17T23:46:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS=() 2024-09-17T23:46:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS=() 2024-09-17T23:46:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS=() 2024-09-17T23:46:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS=() 2024-09-17T23:46:33.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ RET=0 2024-09-17T23:46:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-17T23:46:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary1 2024-09-17T23:46:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-17T23:46:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:46:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-17T23:46:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:46:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:46:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:46:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:33.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-17T23:46:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:46:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:46:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-17T23:46:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-17T23:46:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:46:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-17T23:46:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary1': 2024-09-17T23:46:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:46:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:46:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:46:34.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 19ba7a50c03b 2024-09-17T23:46:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.19ba7a50c03b 2024-09-17T23:46:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:46:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:46:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:46:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:46:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:46:33 2024 2024-09-17T23:46:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:46:33 2024 2024-09-17T23:46:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:46:33 2024 2024-09-17T23:46:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:46:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:46:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 26114b26-116f-40a8-a2cc-e67f85f7b4c4 2024-09-17T23:46:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:46:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:46:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:46:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-17T23:46:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd0 2024-09-17T23:46:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:46:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-17T23:46:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:46:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:46:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:46:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 991dc7bc-7a88-42e4-95ce-d64068ce90ae 2024-09-17T23:46:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:46:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:46:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:46:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:46:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:46:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:46:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary1 2024-09-17T23:46:34.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-17T23:46:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:46:34.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary1 2024-09-17T23:46:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:46:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-17T23:46:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:46:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:46:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:46:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary1 2024-09-17T23:46:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-17T23:46:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary2 2024-09-17T23:46:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-17T23:46:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary1 2024-09-17T23:46:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-17T23:46:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:46:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-17T23:46:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:46:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:46:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:46:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-17T23:46:34.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-17T23:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:34.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:34.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-17T23:46:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:46:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-17T23:46:35.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:46:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 665 2024-09-17T23:46:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:35.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-17T23:46:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary2': 2024-09-17T23:46:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:46:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:46:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:46:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 19cafc65c880 2024-09-17T23:46:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.19cafc65c880 2024-09-17T23:46:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:46:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:46:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:46:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:46:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:46:34 2024 2024-09-17T23:46:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:46:34 2024 2024-09-17T23:46:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:46:34 2024 2024-09-17T23:46:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:46:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:46:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 7e53aedd-f077-49b3-b8e4-40ec80c224e6 2024-09-17T23:46:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:46:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:46:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:46:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-17T23:46:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd1 2024-09-17T23:46:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:46:35.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-17T23:46:35.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:46:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:46:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:46:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: c7c0425f-3920-462e-8a88-8ebd67bdee2c 2024-09-17T23:46:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:46:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:46:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:46:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:46:35.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:46:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:46:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary2 2024-09-17T23:46:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-17T23:46:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:46:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary2 2024-09-17T23:46:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:46:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-17T23:46:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:46:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:46:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary2 2024-09-17T23:46:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-17T23:46:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary3 2024-09-17T23:46:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:46:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-17T23:46:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary2 2024-09-17T23:46:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:35.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:46:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-17T23:46:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:46:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-17T23:46:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:46:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:46:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:46:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:46:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:35.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-17T23:46:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:46:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:46:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-17T23:46:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:46:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:35.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-17T23:46:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:46:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-17T23:46:36.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:46:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 667 2024-09-17T23:46:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:36.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-17T23:46:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary3': 2024-09-17T23:46:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:46:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:46:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:46:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 19d9c6c42b5e 2024-09-17T23:46:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.19d9c6c42b5e 2024-09-17T23:46:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:46:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:46:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:46:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:46:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:46:35 2024 2024-09-17T23:46:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:46:35 2024 2024-09-17T23:46:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:46:35 2024 2024-09-17T23:46:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:46:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:46:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: eb4a6a06-f497-481b-9b07-84f8c49bcbd1 2024-09-17T23:46:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:46:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:46:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:46:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-17T23:46:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd2 2024-09-17T23:46:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:46:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-17T23:46:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:46:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:46:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:46:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: e5f1cd47-1bc5-4290-b851-5d072e9bb87b 2024-09-17T23:46:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:46:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:46:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:46:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:46:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:46:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:46:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary3 2024-09-17T23:46:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-17T23:46:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:46:36.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary3 2024-09-17T23:46:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:46:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-17T23:46:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-17T23:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary4 2024-09-17T23:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary3 2024-09-17T23:46:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:46:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-17T23:46:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:46:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:46:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:46:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary3 2024-09-17T23:46:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-17T23:46:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-17T23:46:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:46:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:46:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:46:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:46:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-17T23:46:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:46:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:46:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-17T23:46:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:46:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-17T23:46:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:46:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-17T23:46:37.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 670 2024-09-17T23:46:37.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:46:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-17T23:46:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary4': 2024-09-17T23:46:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:46:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:46:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:46:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 19e6fa77d8be 2024-09-17T23:46:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.19e6fa77d8be 2024-09-17T23:46:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:46:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:46:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:46:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:46:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:46:36 2024 2024-09-17T23:46:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:46:36 2024 2024-09-17T23:46:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:46:36 2024 2024-09-17T23:46:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:46:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:46:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 305debd5-50a6-46f0-baca-0858687046ee 2024-09-17T23:46:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:46:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:46:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:46:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-17T23:46:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd3 2024-09-17T23:46:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:46:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-17T23:46:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:46:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:46:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:46:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 6c7b4590-052d-41a3-9dbd-f449e100f951 2024-09-17T23:46:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:46:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:46:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:46:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:46:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:46:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:46:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary4 2024-09-17T23:46:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-17T23:46:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:46:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary4 2024-09-17T23:46:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:46:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:46:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-17T23:46:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary5 2024-09-17T23:46:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-17T23:46:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:46:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:46:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-17T23:46:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:46:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:46:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:46:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:46:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-17T23:46:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-17T23:46:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:46:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary4 2024-09-17T23:46:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:46:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary4 2024-09-17T23:46:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-17T23:46:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:46:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:46:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:46:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:46:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-17T23:46:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:37.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:46:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-17T23:46:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:46:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-17T23:46:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 671 2024-09-17T23:46:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:46:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:46:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:46:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-17T23:46:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary5': 2024-09-17T23:46:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:46:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:46:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:46:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 19f31e03b85d 2024-09-17T23:46:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.19f31e03b85d 2024-09-17T23:46:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:46:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:46:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:46:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:46:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:46:37 2024 2024-09-17T23:46:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:46:37 2024 2024-09-17T23:46:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:46:37 2024 2024-09-17T23:46:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:46:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:46:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 945834ef-a945-4296-8d1f-7781cb333d6c 2024-09-17T23:46:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:46:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:46:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:46:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-17T23:46:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd4 2024-09-17T23:46:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:46:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-17T23:46:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:46:38.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:46:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:46:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 8f1116b6-cbf6-4aab-9cbd-b8937e287cf2 2024-09-17T23:46:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:46:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:46:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:46:38.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:46:38.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:46:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:46:38.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary5 2024-09-17T23:46:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-17T23:46:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:46:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary5 2024-09-17T23:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:46:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-17T23:46:38.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:38.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:46:38.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:46:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:46:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-17T23:46:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary5 2024-09-17T23:46:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary6 2024-09-17T23:46:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:46:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-17T23:46:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:46:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary5 2024-09-17T23:46:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:46:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-17T23:46:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:46:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-17T23:46:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:46:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:46:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:46:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:46:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-17T23:46:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:46:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:46:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-17T23:46:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:46:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-17T23:46:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:46:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-17T23:46:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:46:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 674 2024-09-17T23:46:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-17T23:46:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:46:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:46:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary6': 2024-09-17T23:46:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:46:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:46:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:46:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1a064e265383 2024-09-17T23:46:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1a064e265383 2024-09-17T23:46:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:46:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:46:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:46:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:46:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:46:38 2024 2024-09-17T23:46:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:46:38 2024 2024-09-17T23:46:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:46:38 2024 2024-09-17T23:46:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:46:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:46:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: cfe9f926-9ade-41df-8253-26b0f3df4079 2024-09-17T23:46:39.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:46:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:46:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:46:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-17T23:46:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd5 2024-09-17T23:46:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:46:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-17T23:46:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:46:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:46:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:46:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 75239d69-f61c-4c43-9c68-5fbec8e73f47 2024-09-17T23:46:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:46:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:46:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:46:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:46:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:46:39.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:46:39.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary6 2024-09-17T23:46:39.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-17T23:46:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:46:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary6 2024-09-17T23:46:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:46:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-17T23:46:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:46:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:46:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-17T23:46:39.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary6 2024-09-17T23:46:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary7 2024-09-17T23:46:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:46:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-17T23:46:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:46:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:46:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:46:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:46:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary6 2024-09-17T23:46:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-17T23:46:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-17T23:46:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-17T23:46:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:46:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:46:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-17T23:46:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:46:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-17T23:46:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:46:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-17T23:46:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 673 2024-09-17T23:46:40.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:40.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:46:40.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:40.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:46:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:46:40.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 677 2024-09-17T23:46:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:46:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-17T23:46:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary7': 2024-09-17T23:46:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:46:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:46:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:46:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1a172222c8b0 2024-09-17T23:46:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1a172222c8b0 2024-09-17T23:46:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:46:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:46:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:46:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:46:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:46:39 2024 2024-09-17T23:46:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:46:39 2024 2024-09-17T23:46:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:46:39 2024 2024-09-17T23:46:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:46:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:46:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: a9650c79-cb43-4f09-baed-431a05d5bac7 2024-09-17T23:46:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:46:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:46:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:46:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-17T23:46:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd6 2024-09-17T23:46:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:46:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-17T23:46:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:46:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:46:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:46:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: afb63d37-aa6d-4eb3-b91c-6c30a8e4a7f9 2024-09-17T23:46:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:46:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:46:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:46:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:46:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:46:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:46:40.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary7 2024-09-17T23:46:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-17T23:46:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:46:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary7 2024-09-17T23:46:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:46:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-17T23:46:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:46:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:46:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:46:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:46:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:46:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary7 2024-09-17T23:46:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-17T23:46:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary8 2024-09-17T23:46:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-17T23:46:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary7 2024-09-17T23:46:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-17T23:46:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:46:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:46:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-17T23:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-17T23:46:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:46:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:46:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-17T23:46:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:46:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:40.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-17T23:46:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:46:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-17T23:46:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 676 2024-09-17T23:46:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:46:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:46:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:46:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:41.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:46:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 680 2024-09-17T23:46:41.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-17T23:46:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary8': 2024-09-17T23:46:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:46:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:46:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:46:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1a2886123890 2024-09-17T23:46:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1a2886123890 2024-09-17T23:46:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:46:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:46:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:46:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:46:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:46:40 2024 2024-09-17T23:46:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:46:40 2024 2024-09-17T23:46:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:46:40 2024 2024-09-17T23:46:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:46:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:46:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: ab09e279-9786-4088-9ca5-ffb8f13fe226 2024-09-17T23:46:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:46:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:46:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:46:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-17T23:46:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd7 2024-09-17T23:46:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:46:41.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-17T23:46:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:46:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:46:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:46:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: a5caa86e-9277-48f1-b4ad-a6adf969f870 2024-09-17T23:46:41.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:46:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:46:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:46:41.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:46:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:46:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:46:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary8 2024-09-17T23:46:41.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-17T23:46:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:46:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary8 2024-09-17T23:46:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:46:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-17T23:46:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:46:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:46:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:46:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-17T23:46:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary8 2024-09-17T23:46:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary9 2024-09-17T23:46:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:46:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-17T23:46:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:46:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:46:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:46:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary8 2024-09-17T23:46:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-17T23:46:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-17T23:46:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:46:41.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:46:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:46:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:46:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-17T23:46:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:46:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:46:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-17T23:46:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:41.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:46:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-17T23:46:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:46:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-17T23:46:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 679 2024-09-17T23:46:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:46:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:46:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:46:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 684 2024-09-17T23:46:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:46:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:42.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-17T23:46:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary9': 2024-09-17T23:46:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:46:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:46:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:46:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1a3a932779d8 2024-09-17T23:46:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1a3a932779d8 2024-09-17T23:46:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:46:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:46:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:46:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:46:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:46:41 2024 2024-09-17T23:46:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:46:41 2024 2024-09-17T23:46:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:46:41 2024 2024-09-17T23:46:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:46:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:46:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921 2024-09-17T23:46:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:46:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:46:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:46:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-17T23:46:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd8 2024-09-17T23:46:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:46:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-17T23:46:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:46:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:46:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:46:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 44f94563-f248-45b2-8a7a-08d07fbf0730 2024-09-17T23:46:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:46:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:46:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:46:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:46:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:46:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:46:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:42.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary9 2024-09-17T23:46:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-17T23:46:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:46:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary9 2024-09-17T23:46:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:46:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-17T23:46:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:46:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:46:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:46:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:46:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:46:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary9 2024-09-17T23:46:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:46:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-17T23:46:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary10 2024-09-17T23:46:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-17T23:46:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary9 2024-09-17T23:46:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-17T23:46:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:46:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:46:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-17T23:46:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:46:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:46:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:46:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:46:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:46:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-17T23:46:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:46:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:46:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:46:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-17T23:46:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:46:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-17T23:46:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:46:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-17T23:46:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:46:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:43.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:43.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:46:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:46:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:46:43.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:46:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:46:43.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-17T23:46:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 686 2024-09-17T23:46:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:43.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary10': 2024-09-17T23:46:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:46:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:46:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:46:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1a4c49f88236 2024-09-17T23:46:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1a4c49f88236 2024-09-17T23:46:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:46:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:46:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:46:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:46:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:46:42 2024 2024-09-17T23:46:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:46:42 2024 2024-09-17T23:46:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:46:42 2024 2024-09-17T23:46:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:46:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:46:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 085fc961-bab1-4386-91c9-86972b75c274 2024-09-17T23:46:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:46:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:46:43.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:46:43.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-17T23:46:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd9 2024-09-17T23:46:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:46:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-17T23:46:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:46:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:46:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:46:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 9e7d480e-efe3-4434-ac5f-9a08b3c6a309 2024-09-17T23:46:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:46:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:46:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:46:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:46:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:46:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:46:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:46:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary10 2024-09-17T23:46:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-17T23:46:43.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:46:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary10 2024-09-17T23:46:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:46:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-17T23:46:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:46:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:46:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:43.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:46:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:46:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 59403 2024-09-17T23:46:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:46:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary10 2024-09-17T23:46:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:46:43.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary10 2024-09-17T23:46:43.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-17T23:46:44.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:46:44.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-17T23:46:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 682 2024-09-17T23:46:44.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 685 2024-09-17T23:46:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 690 2024-09-17T23:46:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:46:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:46:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:46:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:46:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:46:44.938 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:46:44.935+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-17T23:46:45.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 688 2024-09-17T23:46:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 689 2024-09-17T23:46:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:46:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:46:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:46:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:46:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:46:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:46:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 692 2024-09-17T23:46:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 691 2024-09-17T23:46:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:46:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:46:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:46:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:46:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:46:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:46:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 693 2024-09-17T23:46:47.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:47.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:46:47.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:47.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:47.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:46:47.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:46:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 694 2024-09-17T23:46:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:46:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:46:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:46:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 695 2024-09-17T23:46:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:46:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:46:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:46:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:46:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:46:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 696 2024-09-17T23:46:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 697 2024-09-17T23:46:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:46:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:46:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:46:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:46:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:46:49.757 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-17T23:46:49.751+0000 7f0114255640 -1 reset not still connected to 0x557d0a065ee0 2024-09-17T23:46:49.759 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-17T23:46:49.751+0000 7f0114255640 -1 reset not still connected to 0x557d0b464750 2024-09-17T23:46:49.759 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-17T23:46:49.751+0000 7f0114255640 -1 reset not still connected to 0x557d10f46dd0 2024-09-17T23:46:50.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 700 2024-09-17T23:46:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:46:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:46:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:46:50.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 698 2024-09-17T23:46:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:46:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:46:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:46:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:50.777 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-17T23:46:50.774+0000 7f6acfc5f640 -1 reset not still connected to 0x555c177d80d0 2024-09-17T23:46:50.804 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-17T23:46:50.803+0000 7f6acfc5f640 -1 reset not still connected to 0x555c1e18b380 2024-09-17T23:46:51.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 699 2024-09-17T23:46:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:46:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:46:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:46:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 701 2024-09-17T23:46:51.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 702 2024-09-17T23:46:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:51.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:46:51.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:46:51.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:46:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:46:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:46:52.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:46:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 703 2024-09-17T23:46:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 704 2024-09-17T23:46:52.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 705 2024-09-17T23:46:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 706 2024-09-17T23:46:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:46:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:46:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:53.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:46:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:46:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:46:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 707 2024-09-17T23:46:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:54.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 709 2024-09-17T23:46:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:46:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:46:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:46:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:46:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:46:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:46:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:46:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:46:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 708 2024-09-17T23:46:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:46:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:46:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:46:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:46:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:46:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:46:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 711 2024-09-17T23:46:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 712 2024-09-17T23:46:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 710 2024-09-17T23:46:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:46:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:46:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:46:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:46:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:46:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:46:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:57.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:57.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:46:57.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:57.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:46:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:46:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:46:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:46:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:46:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 714 2024-09-17T23:46:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 713 2024-09-17T23:46:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 715 2024-09-17T23:46:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 716 2024-09-17T23:46:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:46:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:58.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:46:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:46:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:46:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:46:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:46:58.742 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:46:58.740+0000 7f9bfab49640 -1 reset not still connected to 0x55cd2c54e820 2024-09-17T23:46:58.742 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:46:58.740+0000 7f9bfab49640 -1 reset not still connected to 0x55cd30567450 2024-09-17T23:46:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 718 2024-09-17T23:46:59.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:46:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:46:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:46:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:46:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:46:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:46:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:46:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:46:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:46:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:47:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 717 2024-09-17T23:47:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 720 2024-09-17T23:47:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 719 2024-09-17T23:47:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:47:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:47:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:47:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:47:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:47:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:47:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 722 2024-09-17T23:47:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 721 2024-09-17T23:47:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:47:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:47:01.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:47:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:47:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:47:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:47:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 723 2024-09-17T23:47:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 724 2024-09-17T23:47:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:47:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:47:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:47:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 725 2024-09-17T23:47:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 726 2024-09-17T23:47:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:47:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:47:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:47:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:47:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:47:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:47:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:47:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:47:03.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:47:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 727 2024-09-17T23:47:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:47:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:47:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:47:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 730 2024-09-17T23:47:04.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:47:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:47:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:47:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 729 2024-09-17T23:47:05.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:47:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:47:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:47:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:47:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:47:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:47:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 731 2024-09-17T23:47:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 728 2024-09-17T23:47:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 732 2024-09-17T23:47:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:47:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:47:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:47:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:47:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:47:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:47:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 734 2024-09-17T23:47:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 733 2024-09-17T23:47:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:47:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:07.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:47:07.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:47:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:47:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:47:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:47:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:08.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 736 2024-09-17T23:47:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 735 2024-09-17T23:47:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:47:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:47:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:47:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 738 2024-09-17T23:47:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 737 2024-09-17T23:47:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:47:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:47:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:47:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:47:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:47:09.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:47:09.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:09.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:47:09.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:47:09.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:47:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 739 2024-09-17T23:47:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:47:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:47:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:47:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:10.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 741 2024-09-17T23:47:10.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:47:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:47:10.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:47:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:47:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:47:11.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:47:11.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 742 2024-09-17T23:47:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 740 2024-09-17T23:47:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:47:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:47:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:47:11.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:47:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:47:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:47:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:47:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:47:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:47:12.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 744 2024-09-17T23:47:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 743 2024-09-17T23:47:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 746 2024-09-17T23:47:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 745 2024-09-17T23:47:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:47:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:47:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:47:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:13.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:47:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:13.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:47:13.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:47:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 747 2024-09-17T23:47:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 748 2024-09-17T23:47:14.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:14.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:47:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:47:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:47:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:47:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:47:14.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:47:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 749 2024-09-17T23:47:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:47:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:47:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:47:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:47:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:47:15.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:47:15.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 752 2024-09-17T23:47:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 750 2024-09-17T23:47:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 751 2024-09-17T23:47:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:47:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:47:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:47:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:47:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:47:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:47:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:16.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:47:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:47:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:47:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:47:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:47:17.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:47:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 754 2024-09-17T23:47:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 753 2024-09-17T23:47:17.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:17.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:18.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 756 2024-09-17T23:47:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:47:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:47:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:47:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 755 2024-09-17T23:47:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 758 2024-09-17T23:47:19.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:47:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:47:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:47:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:47:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:47:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:47:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:47:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:47:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:47:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 757 2024-09-17T23:47:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 759 2024-09-17T23:47:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:47:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:47:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:47:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:47:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:47:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:47:20.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 761 2024-09-17T23:47:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 762 2024-09-17T23:47:21.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 760 2024-09-17T23:47:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:47:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:47:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:47:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:47:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:47:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:47:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:47:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:47:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:47:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 763 2024-09-17T23:47:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 764 2024-09-17T23:47:22.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 765 2024-09-17T23:47:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:47:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:47:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:47:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:47:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:47:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:47:23.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 766 2024-09-17T23:47:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:47:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:47:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:47:23.491 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:47:23.489+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0e3372b0 2024-09-17T23:47:23.491 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:47:23.489+0000 7f9bfab49640 -1 reset not still connected to 0x55cd2c54e820 2024-09-17T23:47:23.491 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:47:23.489+0000 7f9bfab49640 -1 reset not still connected to 0x55cd30567450 2024-09-17T23:47:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 767 2024-09-17T23:47:24.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 769 2024-09-17T23:47:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:47:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:47:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:47:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:47:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:47:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:47:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 768 2024-09-17T23:47:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 770 2024-09-17T23:47:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:47:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:47:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:47:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:47:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:47:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:47:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 771 2024-09-17T23:47:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 772 2024-09-17T23:47:26.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:47:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:47:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:47:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:47:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:47:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:47:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 773 2024-09-17T23:47:27.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 774 2024-09-17T23:47:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:47:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:27.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:47:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:47:27.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:47:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:47:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:47:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:27.455 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-17T23:47:27.453+0000 7f6acfc5f640 -1 reset not still connected to 0x555c09ea0d00 2024-09-17T23:47:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:47:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:47:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:47:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 775 2024-09-17T23:47:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 776 2024-09-17T23:47:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:47:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:47:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:47:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 777 2024-09-17T23:47:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 778 2024-09-17T23:47:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:47:29.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:29.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:29.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:47:29.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:47:29.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:47:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:47:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:47:30.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 780 2024-09-17T23:47:30.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:47:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:47:30.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:47:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 779 2024-09-17T23:47:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:47:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:47:30.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:47:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:30.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:31.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 781 2024-09-17T23:47:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 782 2024-09-17T23:47:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:47:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:47:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:47:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:47:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:47:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:47:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:47:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:47:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:47:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:47:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:47:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:47:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 783 2024-09-17T23:47:32.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 784 2024-09-17T23:47:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:32.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 785 2024-09-17T23:47:33.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 786 2024-09-17T23:47:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:47:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:47:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:47:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:47:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:47:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:47:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 788 2024-09-17T23:47:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:47:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:47:34.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:34.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:47:34.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:47:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 790 2024-09-17T23:47:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 787 2024-09-17T23:47:35.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 789 2024-09-17T23:47:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:47:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:47:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:47:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:47:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:47:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:47:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 791 2024-09-17T23:47:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:47:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:47:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:47:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:47:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:47:36.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:47:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:47:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:47:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:47:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 792 2024-09-17T23:47:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 793 2024-09-17T23:47:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 794 2024-09-17T23:47:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:37.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 796 2024-09-17T23:47:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 795 2024-09-17T23:47:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:47:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:47:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:47:38.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:47:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:47:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:47:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:47:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:47:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:47:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 797 2024-09-17T23:47:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:39.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 799 2024-09-17T23:47:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:47:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:47:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:47:40.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 800 2024-09-17T23:47:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 798 2024-09-17T23:47:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:47:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:47:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:47:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:47:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:47:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:47:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:47:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:47:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:47:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 801 2024-09-17T23:47:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 803 2024-09-17T23:47:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:47:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:47:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:47:41.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:47:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:47:41.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:47:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 802 2024-09-17T23:47:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 804 2024-09-17T23:47:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:47:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:47:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:47:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:47:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:47:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:47:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 805 2024-09-17T23:47:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:47:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:47:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:47:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 806 2024-09-17T23:47:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:47:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:47:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:47:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:43.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 807 2024-09-17T23:47:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 808 2024-09-17T23:47:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:47:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:47:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:47:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:47:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:47:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:47:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 810 2024-09-17T23:47:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 809 2024-09-17T23:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:47:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:47:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:47:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:47:45.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:47:45.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:46.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 811 2024-09-17T23:47:46.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 812 2024-09-17T23:47:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:47:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:47:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:47:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:47:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:47:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:47:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 814 2024-09-17T23:47:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:47:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:47:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:47:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 813 2024-09-17T23:47:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:47:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:47:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:47:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 815 2024-09-17T23:47:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 816 2024-09-17T23:47:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:47:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:47:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:47:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:47:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:47:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:47:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:49.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 817 2024-09-17T23:47:49.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 818 2024-09-17T23:47:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:49.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:49.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:47:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:47:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:47:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:47:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:47:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:47:50.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 819 2024-09-17T23:47:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 820 2024-09-17T23:47:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:47:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:47:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:47:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:47:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:47:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:47:50.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 821 2024-09-17T23:47:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 822 2024-09-17T23:47:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:47:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:47:51.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:47:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:47:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:47:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:47:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:47:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:47:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:47:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 823 2024-09-17T23:47:52.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:47:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:47:52.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:47:52.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 824 2024-09-17T23:47:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 825 2024-09-17T23:47:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:47:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:47:53.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:47:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:47:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:47:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:47:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 828 2024-09-17T23:47:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 826 2024-09-17T23:47:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 827 2024-09-17T23:47:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:47:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:47:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:47:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:47:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:47:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:47:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 829 2024-09-17T23:47:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 830 2024-09-17T23:47:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:47:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:47:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:47:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:47:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:47:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:47:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 831 2024-09-17T23:47:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 832 2024-09-17T23:47:56.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:47:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:56.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:47:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:47:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:47:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:47:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:47:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:47:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:47:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:47:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 833 2024-09-17T23:47:57.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:57.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:47:57.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:57.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:57.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:47:57.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:47:57.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 834 2024-09-17T23:47:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 835 2024-09-17T23:47:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:47:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:47:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:47:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:47:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:47:58.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:47:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 838 2024-09-17T23:47:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 836 2024-09-17T23:47:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:47:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:47:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:47:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:47:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:47:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:47:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:47:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:47:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:47:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:48:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 837 2024-09-17T23:48:00.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 839 2024-09-17T23:48:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:48:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:48:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:48:00.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 840 2024-09-17T23:48:01.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 842 2024-09-17T23:48:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 841 2024-09-17T23:48:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:48:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:48:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:48:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:48:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:48:01.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:48:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:48:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:48:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:48:02.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:48:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:48:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:48:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 843 2024-09-17T23:48:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 844 2024-09-17T23:48:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 845 2024-09-17T23:48:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 846 2024-09-17T23:48:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:48:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:48:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:48:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:48:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:48:03.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:48:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 849 2024-09-17T23:48:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:04.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:48:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:48:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:48:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:48:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:48:04.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:48:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:48:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:48:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:48:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 847 2024-09-17T23:48:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 848 2024-09-17T23:48:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:48:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:48:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:48:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 852 2024-09-17T23:48:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:48:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:48:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:48:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:48:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:48:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:48:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 850 2024-09-17T23:48:06.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:06.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:48:06.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:06.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:06.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:48:06.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:48:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:07.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 854 2024-09-17T23:48:07.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 851 2024-09-17T23:48:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 853 2024-09-17T23:48:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:48:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:48:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:48:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:07.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:08.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 855 2024-09-17T23:48:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 856 2024-09-17T23:48:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:48:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:48:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:48:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:48:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:48:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:48:08.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:08.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:48:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:48:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:48:09.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 857 2024-09-17T23:48:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 858 2024-09-17T23:48:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:48:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:48:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:48:10.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 860 2024-09-17T23:48:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 859 2024-09-17T23:48:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:48:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:48:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:48:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:48:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:48:10.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:48:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:48:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:48:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:48:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 861 2024-09-17T23:48:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 863 2024-09-17T23:48:11.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:48:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:48:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:48:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:48:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:48:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:48:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 864 2024-09-17T23:48:12.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:12.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:48:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:48:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:48:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:48:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:48:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:48:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 862 2024-09-17T23:48:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 866 2024-09-17T23:48:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 865 2024-09-17T23:48:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:48:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:48:13.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:48:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:48:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:48:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:48:13.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:14.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 868 2024-09-17T23:48:14.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 867 2024-09-17T23:48:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:48:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:48:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:48:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:14.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:48:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:48:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:48:15.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 869 2024-09-17T23:48:15.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 870 2024-09-17T23:48:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:48:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:48:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:48:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 871 2024-09-17T23:48:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 872 2024-09-17T23:48:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:48:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:48:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:48:16.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:48:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:48:16.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:48:16.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:48:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:48:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:48:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 873 2024-09-17T23:48:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:48:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:48:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:48:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:48:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:48:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:48:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:17.428 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:48:17.426+0000 7f706e974640 -1 reset not still connected to 0x55df28279450 2024-09-17T23:48:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 874 2024-09-17T23:48:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 875 2024-09-17T23:48:18.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 876 2024-09-17T23:48:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:48:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:48:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:48:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:48:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:48:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:48:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:19.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 878 2024-09-17T23:48:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:48:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:48:19.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:48:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:19.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:48:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:48:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:48:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 877 2024-09-17T23:48:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 880 2024-09-17T23:48:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 879 2024-09-17T23:48:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:48:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:48:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:48:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:48:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:48:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:48:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 882 2024-09-17T23:48:21.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 881 2024-09-17T23:48:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:48:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:48:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:48:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:48:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:48:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:48:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:48:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:48:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:48:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 883 2024-09-17T23:48:22.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:23.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 886 2024-09-17T23:48:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 884 2024-09-17T23:48:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 885 2024-09-17T23:48:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:48:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:48:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:48:23.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:48:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:48:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:48:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:48:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 887 2024-09-17T23:48:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 890 2024-09-17T23:48:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:48:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:48:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:48:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:48:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:48:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:48:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 889 2024-09-17T23:48:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 888 2024-09-17T23:48:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:48:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:48:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:48:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:26.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 892 2024-09-17T23:48:26.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 891 2024-09-17T23:48:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:48:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:48:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:48:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:48:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:48:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:48:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:48:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:48:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:48:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:48:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:48:27.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:48:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:48:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:48:27.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:48:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 894 2024-09-17T23:48:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 893 2024-09-17T23:48:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 896 2024-09-17T23:48:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 895 2024-09-17T23:48:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:48:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:28.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:48:28.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:48:28.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:28.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:48:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:48:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:48:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 897 2024-09-17T23:48:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 898 2024-09-17T23:48:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:48:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:48:29.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:48:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:48:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:48:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:48:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 899 2024-09-17T23:48:30.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 900 2024-09-17T23:48:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:48:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:48:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:48:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:48:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:48:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:48:31.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 902 2024-09-17T23:48:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:48:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:48:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:48:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:48:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:48:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:48:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:48:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:48:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:48:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:48:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:48:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:48:32.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 903 2024-09-17T23:48:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 901 2024-09-17T23:48:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 906 2024-09-17T23:48:32.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:32.639 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-17T23:48:32.637+0000 7f0114255640 -1 reset not still connected to 0x557d0b464750 2024-09-17T23:48:33.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 904 2024-09-17T23:48:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:48:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:48:33.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:48:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:48:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:48:33.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:48:33.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 907 2024-09-17T23:48:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 908 2024-09-17T23:48:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 905 2024-09-17T23:48:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:34.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:34.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:48:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:48:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:48:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 910 2024-09-17T23:48:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 909 2024-09-17T23:48:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:35.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:48:35.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:48:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:48:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:48:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:48:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:48:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:48:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:48:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 911 2024-09-17T23:48:36.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 913 2024-09-17T23:48:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:48:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:48:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:48:37.378 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:48:37.376+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4dd0 2024-09-17T23:48:37.378 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:48:37.377+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb5790 2024-09-17T23:48:37.379 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:48:37.377+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ea270 2024-09-17T23:48:37.382 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:48:37.380+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-17T23:48:37.382 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:48:37.380+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-17T23:48:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 912 2024-09-17T23:48:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 914 2024-09-17T23:48:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:48:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:48:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:48:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:48:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:48:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:48:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:48:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:48:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:48:38.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 915 2024-09-17T23:48:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:48:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:48:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:48:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 917 2024-09-17T23:48:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:48:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:48:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:48:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 916 2024-09-17T23:48:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 918 2024-09-17T23:48:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:48:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:48:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:48:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:48:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:48:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:48:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 920 2024-09-17T23:48:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 919 2024-09-17T23:48:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:48:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:48:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:48:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:48:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:48:40.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:48:40.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:40.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:48:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:48:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:48:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 921 2024-09-17T23:48:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 922 2024-09-17T23:48:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:48:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:48:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:48:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:41.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 923 2024-09-17T23:48:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 924 2024-09-17T23:48:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:48:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:42.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:48:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:48:42.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:48:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:48:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:48:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:42.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 925 2024-09-17T23:48:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 926 2024-09-17T23:48:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:48:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:48:43.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:48:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:48:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:48:43.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:48:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:43.698 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:48:43.695+0000 7fc275b74640 -1 reset not still connected to 0x559cc0471d40 2024-09-17T23:48:43.698 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:48:43.695+0000 7fc275b74640 -1 reset not still connected to 0x559cc04b3ba0 2024-09-17T23:48:43.698 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:48:43.696+0000 7fc275b74640 -1 reset not still connected to 0x559cc08b4a90 2024-09-17T23:48:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 928 2024-09-17T23:48:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:48:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:48:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:48:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:48:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:48:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:48:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 927 2024-09-17T23:48:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 929 2024-09-17T23:48:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:48:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:48:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:48:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:48:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:48:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:48:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 931 2024-09-17T23:48:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 930 2024-09-17T23:48:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 932 2024-09-17T23:48:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:48:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:48:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:48:46.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:48:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:48:46.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:48:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:48:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:48:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:48:47.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 934 2024-09-17T23:48:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 933 2024-09-17T23:48:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 936 2024-09-17T23:48:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 935 2024-09-17T23:48:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:48:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:48:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:48:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:48:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:48:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:48:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 937 2024-09-17T23:48:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:49.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 938 2024-09-17T23:48:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:48:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:48:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:48:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:48:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:48:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:48:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:48:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:48:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:48:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 939 2024-09-17T23:48:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:50.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 941 2024-09-17T23:48:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:48:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:48:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:48:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:48:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:48:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:48:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 940 2024-09-17T23:48:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 942 2024-09-17T23:48:51.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:48:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:48:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:48:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:48:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:48:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:48:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:48:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:48:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:48:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:48:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 943 2024-09-17T23:48:52.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 944 2024-09-17T23:48:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:48:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:48:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:48:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:48:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:48:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:48:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 945 2024-09-17T23:48:53.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 946 2024-09-17T23:48:53.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:53.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 947 2024-09-17T23:48:54.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 948 2024-09-17T23:48:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:48:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:48:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:48:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:48:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:48:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:48:55.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 950 2024-09-17T23:48:55.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 949 2024-09-17T23:48:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:48:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:48:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:48:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:48:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:48:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:48:56.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 952 2024-09-17T23:48:56.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 951 2024-09-17T23:48:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:48:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:48:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:48:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:48:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:48:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:48:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:48:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:48:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:48:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:48:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:48:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:48:57.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 953 2024-09-17T23:48:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 954 2024-09-17T23:48:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:58.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 955 2024-09-17T23:48:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 956 2024-09-17T23:48:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:48:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:48:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:48:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:48:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:48:58.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:48:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 958 2024-09-17T23:48:59.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 957 2024-09-17T23:48:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:48:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:48:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:48:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:48:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:48:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:48:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:48:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:48:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:48:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:49:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 959 2024-09-17T23:49:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 960 2024-09-17T23:49:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:49:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:49:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:49:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:49:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:49:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:49:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:49:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:49:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:49:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:49:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:49:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:49:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:49:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:49:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 961 2024-09-17T23:49:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 962 2024-09-17T23:49:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:49:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:49:01.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:49:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:49:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:49:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:49:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:49:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:49:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:49:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:49:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:49:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:49:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:49:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:49:02.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 964 2024-09-17T23:49:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 963 2024-09-17T23:49:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:49:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:49:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:49:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:49:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:49:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:49:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:49:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:49:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:49:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:49:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:49:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:49:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:49:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:49:03.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 966 2024-09-17T23:49:03.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 965 2024-09-17T23:49:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:49:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:49:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:49:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 59535 2024-09-17T23:49:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:49:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:49:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 967 2024-09-17T23:49:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 968 2024-09-17T23:49:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:49:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 59667 2024-09-17T23:49:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:49:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:49:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:49:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:49:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:49:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:49:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:49:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:49:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 969 2024-09-17T23:49:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:49:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:49:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:49:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:49:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:49:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:49:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:49:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:49:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 59799 2024-09-17T23:49:06.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 970 2024-09-17T23:49:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:49:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 59951 2024-09-17T23:49:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:49:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:49:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:49:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:49:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:49:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:49:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:49:07.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 971 2024-09-17T23:49:07.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:49:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:49:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:49:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:49:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:49:07.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:49:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:49:07.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 60103 2024-09-17T23:49:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:49:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 972 2024-09-17T23:49:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:49:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 60256 2024-09-17T23:49:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:49:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 973 2024-09-17T23:49:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:49:09.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:49:09.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 60409 2024-09-17T23:49:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:49:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 60562 2024-09-17T23:49:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:49:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 60734 2024-09-17T23:49:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-17T23:49:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:49:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 59404 2024-09-17T23:51:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:51:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:51:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:51:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 59536 2024-09-17T23:51:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:51:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:51:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:51:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 59668 2024-09-17T23:51:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:51:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:51:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:51:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 59800 2024-09-17T23:51:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:51:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:51:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:51:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 59952 2024-09-17T23:51:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:51:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:51:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:51:38.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 60104 2024-09-17T23:51:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:51:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:51:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:51:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 60257 2024-09-17T23:51:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:51:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:51:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:51:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 60410 2024-09-17T23:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:51:41.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:51:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:51:41.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 60563 2024-09-17T23:51:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:51:42.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:51:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:51:42.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 60735 2024-09-17T23:51:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:51:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:51:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-17T23:51:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:51:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:51:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:51:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 1 2024-09-17T23:51:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd0 2024-09-17T23:51:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:51:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary1 2024-09-17T23:51:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary1 2024-09-17T23:51:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:51:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary1 2024-09-17T23:51:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:51:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 2 2024-09-17T23:51:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd1 2024-09-17T23:51:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:51:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary2 2024-09-17T23:51:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary2 2024-09-17T23:51:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:51:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary2 2024-09-17T23:51:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:51:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 3 2024-09-17T23:51:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd2 2024-09-17T23:51:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary3 2024-09-17T23:51:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary3 2024-09-17T23:51:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:51:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:51:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary3 2024-09-17T23:51:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 4 2024-09-17T23:51:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:51:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd3 2024-09-17T23:51:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary4 2024-09-17T23:51:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary4 2024-09-17T23:51:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:51:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary4 2024-09-17T23:51:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:51:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:51:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 5 2024-09-17T23:51:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:51:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd4 2024-09-17T23:51:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary5 2024-09-17T23:51:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:51:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary5 2024-09-17T23:51:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 6 2024-09-17T23:51:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:51:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary5 2024-09-17T23:51:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd5 2024-09-17T23:51:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary6 2024-09-17T23:51:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:51:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:51:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary6 2024-09-17T23:51:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 7 2024-09-17T23:51:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:51:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary6 2024-09-17T23:51:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd6 2024-09-17T23:51:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary7 2024-09-17T23:51:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary7 2024-09-17T23:51:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:51:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:51:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary7 2024-09-17T23:51:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:51:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:51:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:51:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:51:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:51:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 67640 2024-09-17T23:51:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 10 2024-09-17T23:51:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd9 2024-09-17T23:51:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary10 2024-09-17T23:51:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary10 2024-09-17T23:51:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:51:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary10 2024-09-17T23:51:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 9 2024-09-17T23:51:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd8 2024-09-17T23:51:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary9 2024-09-17T23:51:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary9 2024-09-17T23:51:44.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:51:44.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary9 2024-09-17T23:51:44.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 8 2024-09-17T23:51:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd7 2024-09-17T23:51:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary8 2024-09-17T23:51:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary8 2024-09-17T23:51:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:51:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary8 2024-09-17T23:51:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-17T23:51:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:51:44.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:51:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-17T23:51:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-17T23:51:44.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:51:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-17T23:51:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:51:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-17T23:51:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:51:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-17T23:51:44.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:51:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-17T23:51:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:51:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:51:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-17T23:51:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:51:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-17T23:51:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:51:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-17T23:52:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=ee5721ff79d7948ab61bdfb0b5aea852 2024-09-17T23:52:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-17T23:52:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=dda71afca0613477222ef627c266aa6f 2024-09-17T23:52:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-17T23:52:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-17T23:52:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:52:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-17T23:52:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-17T23:52:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:52:14.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-17T23:52:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=b543ef7e67f11e527db7356d830a8105 2024-09-17T23:52:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-17T23:52:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-17T23:52:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:52:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-17T23:52:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=d07fb0a7e462286b53b4004e3b1e15ee 2024-09-17T23:52:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-17T23:52:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-17T23:52:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:52:14.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-17T23:52:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=9e2259963f44bccf6e7bf26b1994ef24 2024-09-17T23:52:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-17T23:52:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=2ac1a348b0ba0fbb5fc1ff9e14933d66 2024-09-17T23:52:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-17T23:52:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=85f8d71a7a6755f879fd0e398f4b429c 2024-09-17T23:52:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-17T23:52:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-17T23:52:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:52:15.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-17T23:52:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=242ae467b4a4fd1e04a7466caaa45a7a 2024-09-17T23:52:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-17T23:52:15.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-17T23:52:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:52:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-17T23:52:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-17T23:52:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:52:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-17T23:52:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-17T23:52:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:52:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-17T23:52:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:52:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:52:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-17T23:52:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:52:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-17T23:52:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:52:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=9a91c277bab39f0a64d7ba6786da12c2 2024-09-17T23:52:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-17T23:52:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-17T23:52:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:52:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-17T23:52:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=d62e892e624621370b44a6f8cfd91063 2024-09-17T23:52:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-17T23:52:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-17T23:52:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:52:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-17T23:52:16.387 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:52:16.385+0000 7fc275b74640 -1 reset not still connected to 0x559cbebc1520 2024-09-17T23:52:16.388 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:52:16.386+0000 7fc275b74640 -1 reset not still connected to 0x559cc04b3ba0 2024-09-17T23:52:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:52:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:52:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:52:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-17T23:52:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:52:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:16.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-17T23:52:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-17T23:52:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:52:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-17T23:52:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:52:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-17T23:52:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:52:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-17T23:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-17T23:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-17T23:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 945834ef-a945-4296-8d1f-7781cb333d6c 2024-09-17T23:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:12 2024-09-17T23:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:16' 2024-09-17T23:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-17T23:52:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-17T23:52:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-17T23:52:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 26114b26-116f-40a8-a2cc-e67f85f7b4c4 2024-09-17T23:52:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:52:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:52:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:12 2024-09-17T23:52:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:52:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:52:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:16' 2024-09-17T23:52:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-17T23:52:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-17T23:52:17.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:52:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:52:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:52:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:52:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:52:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-17T23:52:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:52:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-17T23:52:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:52:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-17T23:52:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-17T23:52:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:52:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-17T23:52:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-17T23:52:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:52:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:17.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-17T23:52:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-17T23:52:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:52:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-17T23:52:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-17T23:52:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:52:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:52:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-17T23:52:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:52:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-17T23:52:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ab09e279-9786-4088-9ca5-ffb8f13fe226 2024-09-17T23:52:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17 2024-09-17T23:52:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:52:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17' 2024-09-17T23:52:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:52:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-17T23:52:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 085fc961-bab1-4386-91c9-86972b75c274 2024-09-17T23:52:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17 2024-09-17T23:52:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:52:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17' 2024-09-17T23:52:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:52:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-17 23:52:17' 2024-09-17T23:52:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:52:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:17' 2024-09-17T23:52:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-17 23:52:17' 2024-09-17T23:52:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:52:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:17' 2024-09-17T23:52:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-17T23:52:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 305debd5-50a6-46f0-baca-0858687046ee 2024-09-17T23:52:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17 2024-09-17T23:52:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:52:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17' 2024-09-17T23:52:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:52:17.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-17 23:52:17' 2024-09-17T23:52:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:52:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:17' 2024-09-17T23:52:17.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2592768.0,"last_snapshot_bytes":5046272,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616940,"remote_snapshot_timestamp":1726616940,"replay_state":"idle"}2024-09-17 23:52:17image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1093632.0,"last_snapshot_bytes":2043904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616941,"remote_snapshot_timestamp":1726616941,"replay_state":"idle"}2024-09-17 23:52:17image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2625536.0,"last_snapshot_bytes":5115904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616944,"remote_snapshot_timestamp":1726616944,"replay_state":"idle"}2024-09-17 23:52:17image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2570240.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616945,"remote_snapshot_timestamp":1726616945,"replay_state":"idle"}2024-09-17 23:52:17image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":942080.0,"last_snapshot_bytes":1740800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616947,"remote_snapshot_timestamp":1726616947,"replay_state":"idle"}2024-09-17 23:52:17' 2024-09-17T23:52:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-17T23:52:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-17T23:52:18.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-17T23:52:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:52:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2592768.0,"last_snapshot_bytes":5046272,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616940,"remote_snapshot_timestamp":1726616940,"replay_state":"idle"}2024-09-17 23:52:17image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1093632.0,"last_snapshot_bytes":2043904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616941,"remote_snapshot_timestamp":1726616941,"replay_state":"idle"}2024-09-17 23:52:17image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2625536.0,"last_snapshot_bytes":5115904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616944,"remote_snapshot_timestamp":1726616944,"replay_state":"idle"}2024-09-17 23:52:17image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2570240.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616945,"remote_snapshot_timestamp":1726616945,"replay_state":"idle"}2024-09-17 23:52:17image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":942080.0,"last_snapshot_bytes":1740800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616947,"remote_snapshot_timestamp":1726616947,"replay_state":"idle"}2024-09-17 23:52:17' 2024-09-17T23:52:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-17T23:52:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-17T23:52:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2592768.0,"last_snapshot_bytes":5046272,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616940,"remote_snapshot_timestamp":1726616940,"replay_state":"idle"}2024-09-17 23:52:17image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1093632.0,"last_snapshot_bytes":2043904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616941,"remote_snapshot_timestamp":1726616941,"replay_state":"idle"}2024-09-17 23:52:17image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2625536.0,"last_snapshot_bytes":5115904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616944,"remote_snapshot_timestamp":1726616944,"replay_state":"idle"}2024-09-17 23:52:17image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2570240.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616945,"remote_snapshot_timestamp":1726616945,"replay_state":"idle"}2024-09-17 23:52:17image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":942080.0,"last_snapshot_bytes":1740800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726616947,"remote_snapshot_timestamp":1726616947,"replay_state":"idle"}2024-09-17 23:52:17' 2024-09-17T23:52:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-17T23:52:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-17T23:52:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:52:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-17T23:52:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-17T23:52:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:52:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-17T23:52:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:52:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-17T23:52:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-17T23:52:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:52:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-17T23:52:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-17T23:52:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-17T23:52:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:52:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:52:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-17T23:52:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-17T23:52:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-17T23:52:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:52:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-17T23:52:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-17T23:52:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: cfe9f926-9ade-41df-8253-26b0f3df4079 2024-09-17T23:52:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17 2024-09-17T23:52:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:52:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:52:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:18' 2024-09-17T23:52:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:52:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-17 23:52:17' 2024-09-17T23:52:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:52:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:17' 2024-09-17T23:52:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-17T23:52:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: eb4a6a06-f497-481b-9b07-84f8c49bcbd1 2024-09-17T23:52:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17 2024-09-17T23:52:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:52:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:52:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:18' 2024-09-17T23:52:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:52:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-17 23:52:17' 2024-09-17T23:52:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:52:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:17' 2024-09-17T23:52:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-17T23:52:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:52:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-17T23:52:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-17T23:52:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 7e53aedd-f077-49b3-b8e4-40ec80c224e6 2024-09-17T23:52:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17 2024-09-17T23:52:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:52:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:52:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:18' 2024-09-17T23:52:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:18.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:52:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-17T23:52:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921 2024-09-17T23:52:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17 2024-09-17T23:52:18.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:52:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:52:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:18' 2024-09-17T23:52:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:52:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:18.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-17 23:52:17' 2024-09-17T23:52:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:52:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:17' 2024-09-17T23:52:18.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:18.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-17 23:52:17' 2024-09-17T23:52:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:52:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:17' 2024-09-17T23:52:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:18.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-17T23:52:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:52:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:18.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-17T23:52:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18' 2024-09-17T23:52:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-17T23:52:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-17T23:52:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-17T23:52:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:52:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-17T23:52:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 945834ef-a945-4296-8d1f-7781cb333d6c 2024-09-17T23:52:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17 2024-09-17T23:52:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:52:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17' 2024-09-17T23:52:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:52:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-17 23:52:17' 2024-09-17T23:52:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:52:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:17' 2024-09-17T23:52:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:18.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18' 2024-09-17T23:52:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-17T23:52:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-17T23:52:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-17T23:52:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:52:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-17T23:52:18.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 26114b26-116f-40a8-a2cc-e67f85f7b4c4 2024-09-17T23:52:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17 2024-09-17T23:52:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:52:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17' 2024-09-17T23:52:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:52:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18' 2024-09-17T23:52:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:18.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-17T23:52:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-17T23:52:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-17 23:52:17' 2024-09-17T23:52:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:52:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:17' 2024-09-17T23:52:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-17T23:52:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:52:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18' 2024-09-17T23:52:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-17T23:52:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-17T23:52:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:18.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-17T23:52:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:52:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18' 2024-09-17T23:52:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-17T23:52:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-17T23:52:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-17T23:52:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:52:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-17T23:52:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a9650c79-cb43-4f09-baed-431a05d5bac7 2024-09-17T23:52:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17 2024-09-17T23:52:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:52:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:52:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:18' 2024-09-17T23:52:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:52:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:18.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-17 23:52:17' 2024-09-17T23:52:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:52:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:17' 2024-09-17T23:52:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18' 2024-09-17T23:52:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-17T23:52:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-17T23:52:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-17T23:52:18.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:52:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:18.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:52:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:52:18' 2024-09-17T23:52:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-17T23:52:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-17T23:52:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-17T23:52:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:52:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:19.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-17T23:52:19.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:52:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-17T23:52:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-17T23:52:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:52:19.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:19.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:19.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-17T23:52:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-17T23:52:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:52:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-17T23:52:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-17T23:52:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 085fc961-bab1-4386-91c9-86972b75c274 2024-09-17T23:52:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17 2024-09-17T23:52:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:52:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17' 2024-09-17T23:52:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:52:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-17 23:52:17' 2024-09-17T23:52:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:52:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:17' 2024-09-17T23:52:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-17T23:52:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ab09e279-9786-4088-9ca5-ffb8f13fe226 2024-09-17T23:52:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17 2024-09-17T23:52:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:52:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17' 2024-09-17T23:52:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:52:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-17T23:52:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 305debd5-50a6-46f0-baca-0858687046ee 2024-09-17T23:52:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17 2024-09-17T23:52:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:52:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17' 2024-09-17T23:52:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:52:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-17 23:52:17' 2024-09-17T23:52:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:52:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:17' 2024-09-17T23:52:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:19.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-17 23:52:17' 2024-09-17T23:52:19.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:52:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:17' 2024-09-17T23:52:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17' 2024-09-17T23:52:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:19.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-17T23:52:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:19.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-17T23:52:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:19.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:19.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-17T23:52:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:52:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:52:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-17T23:52:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17' 2024-09-17T23:52:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-17T23:52:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:19.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-17T23:52:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17' 2024-09-17T23:52:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-17T23:52:19.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:19.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-17T23:52:19.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:52:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:52:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-17T23:52:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-17T23:52:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-17T23:52:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:52:19.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:52:19.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-17T23:52:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-17T23:52:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:52:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-17T23:52:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-17T23:52:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: cfe9f926-9ade-41df-8253-26b0f3df4079 2024-09-17T23:52:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:52:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:18 2024-09-17T23:52:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:52:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17' 2024-09-17T23:52:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:52:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:19.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:18' 2024-09-17T23:52:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-17 23:52:18' 2024-09-17T23:52:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:52:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:18' 2024-09-17T23:52:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-17T23:52:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:52:19.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-17T23:52:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-17T23:52:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:52:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-17T23:52:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-17T23:52:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:52:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-17T23:52:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-17T23:52:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: eb4a6a06-f497-481b-9b07-84f8c49bcbd1 2024-09-17T23:52:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:52:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:18 2024-09-17T23:52:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:52:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17' 2024-09-17T23:52:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:52:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-17T23:52:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:52:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:18' 2024-09-17T23:52:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-17 23:52:18' 2024-09-17T23:52:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:52:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:18' 2024-09-17T23:52:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-17T23:52:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17' 2024-09-17T23:52:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-17T23:52:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:18' 2024-09-17T23:52:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-17T23:52:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-17T23:52:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:52:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:52:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-17T23:52:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-17T23:52:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:52:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-17T23:52:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-17T23:52:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 7e53aedd-f077-49b3-b8e4-40ec80c224e6 2024-09-17T23:52:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:52:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:18 2024-09-17T23:52:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:52:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17' 2024-09-17T23:52:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:52:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-17T23:52:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921 2024-09-17T23:52:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:52:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:18 2024-09-17T23:52:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:52:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17' 2024-09-17T23:52:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:18' 2024-09-17T23:52:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-17 23:52:18' 2024-09-17T23:52:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:52:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:18' 2024-09-17T23:52:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:52:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:18' 2024-09-17T23:52:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-17 23:52:18' 2024-09-17T23:52:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:52:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:18' 2024-09-17T23:52:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-17T23:52:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 945834ef-a945-4296-8d1f-7781cb333d6c 2024-09-17T23:52:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17 2024-09-17T23:52:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:52:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17' 2024-09-17T23:52:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:19.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:52:19.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-17 23:52:17' 2024-09-17T23:52:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:52:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:17' 2024-09-17T23:52:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-17T23:52:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:52:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-17T23:52:20.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17' 2024-09-17T23:52:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-17T23:52:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-17T23:52:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 26114b26-116f-40a8-a2cc-e67f85f7b4c4 2024-09-17T23:52:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17 2024-09-17T23:52:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:52:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17' 2024-09-17T23:52:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:18' 2024-09-17T23:52:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-17T23:52:20.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:20.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:52:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-17T23:52:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:52:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:52:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-17T23:52:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-17 23:52:17' 2024-09-17T23:52:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:52:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:17' 2024-09-17T23:52:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-17T23:52:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a9650c79-cb43-4f09-baed-431a05d5bac7 2024-09-17T23:52:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:52:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:18 2024-09-17T23:52:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:52:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:52:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:52:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:17' 2024-09-17T23:52:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:52:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:52:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:52:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:18' 2024-09-17T23:52:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-17 23:52:18' 2024-09-17T23:52:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:52:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:52:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:18' 2024-09-17T23:52:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17' 2024-09-17T23:52:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17' 2024-09-17T23:52:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-17T23:52:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-17T23:52:20.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:18' 2024-09-17T23:52:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:18' 2024-09-17T23:52:20.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-17T23:52:20.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-17T23:52:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-17T23:52:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:52:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:52:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-17T23:52:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-17T23:52:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:52:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:52:20.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-17T23:52:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17' 2024-09-17T23:52:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-17T23:52:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-17T23:52:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:20.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-17T23:52:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:52:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:52:20.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-17T23:52:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:52:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:52:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:52:20.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:52:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:52:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:52:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:52:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-17T23:52:20.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:52:20.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:52:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-17T23:52:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:52:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:52:20.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:52:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:52:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-17T23:52:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-17T23:52:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:52:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:52:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-17T23:52:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17' 2024-09-17T23:52:20.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-17T23:52:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:17' 2024-09-17T23:52:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-17T23:52:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-17T23:52:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:52:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:52:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-17T23:52:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary10085fc961-bab1-4386-91c9-86972b75c274up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary4305debd5-50a6-46f0-baca-0858687046eeup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:52:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:52:18cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:52:17' 2024-09-17T23:52:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-17T23:52:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:18' 2024-09-17T23:52:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-17T23:52:20.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:52:20.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:52:20.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:52:20.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:52:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-17T23:52:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:52:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:52:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-17T23:52:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd1 2024-09-17T23:52:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-17T23:52:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:52:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd0 2024-09-17T23:52:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-17T23:52:20.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:52:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd3 2024-09-17T23:52:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:52:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-17T23:52:20.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd2 2024-09-17T23:52:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-17T23:52:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:52:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd4 2024-09-17T23:52:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-17T23:52:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:52:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:52:21.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:52:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:52:21.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:52:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:52:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:52:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:52:21.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-17T23:52:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:52:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:52:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:52:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:52:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:52:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:52:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-17T23:52:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-17T23:52:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-17T23:52:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:52:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:52:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-17T23:52:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd5 2024-09-17T23:52:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:52:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-17T23:52:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd7 2024-09-17T23:52:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:52:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-17T23:52:21.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd8 2024-09-17T23:52:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd9 2024-09-17T23:52:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-17T23:52:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd6 2024-09-17T23:52:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-17T23:52:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:52:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:52:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:52:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-17T23:52:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=ee5721ff79d7948ab61bdfb0b5aea852 2024-09-17T23:52:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-17T23:52:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ ee5721ff79d7948ab61bdfb0b5aea852 != \e\e\5\7\2\1\f\f\7\9\d\7\9\4\8\a\b\6\1\b\d\f\b\0\b\5\a\e\a\8\5\2 ]] 2024-09-17T23:52:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=b543ef7e67f11e527db7356d830a8105 2024-09-17T23:52:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-17T23:52:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ b543ef7e67f11e527db7356d830a8105 != \b\5\4\3\e\f\7\e\6\7\f\1\1\e\5\2\7\d\b\7\3\5\6\d\8\3\0\a\8\1\0\5 ]] 2024-09-17T23:52:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=9a91c277bab39f0a64d7ba6786da12c2 2024-09-17T23:52:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-17T23:52:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 9a91c277bab39f0a64d7ba6786da12c2 != \9\a\9\1\c\2\7\7\b\a\b\3\9\f\0\a\6\4\d\7\b\a\6\7\8\6\d\a\1\2\c\2 ]] 2024-09-17T23:52:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=242ae467b4a4fd1e04a7466caaa45a7a 2024-09-17T23:52:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-17T23:52:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 242ae467b4a4fd1e04a7466caaa45a7a != \2\4\2\a\e\4\6\7\b\4\a\4\f\d\1\e\0\4\a\7\4\6\6\c\a\a\a\4\5\a\7\a ]] 2024-09-17T23:52:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=d07fb0a7e462286b53b4004e3b1e15ee 2024-09-17T23:52:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-17T23:52:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ d07fb0a7e462286b53b4004e3b1e15ee != \d\0\7\f\b\0\a\7\e\4\6\2\2\8\6\b\5\3\b\4\0\0\4\e\3\b\1\e\1\5\e\e ]] 2024-09-17T23:52:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=9e2259963f44bccf6e7bf26b1994ef24 2024-09-17T23:52:52.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-17T23:52:52.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 9e2259963f44bccf6e7bf26b1994ef24 != \9\e\2\2\5\9\9\6\3\f\4\4\b\c\c\f\6\e\7\b\f\2\6\b\1\9\9\4\e\f\2\4 ]] 2024-09-17T23:52:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=d62e892e624621370b44a6f8cfd91063 2024-09-17T23:52:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-17T23:52:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=2ac1a348b0ba0fbb5fc1ff9e14933d66 2024-09-17T23:52:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-17T23:52:53.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ d62e892e624621370b44a6f8cfd91063 != \d\6\2\e\8\9\2\e\6\2\4\6\2\1\3\7\0\b\4\4\a\6\f\8\c\f\d\9\1\0\6\3 ]] 2024-09-17T23:52:53.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 2ac1a348b0ba0fbb5fc1ff9e14933d66 != \2\a\c\1\a\3\4\8\b\0\b\a\0\f\b\b\5\f\c\1\f\f\9\e\1\4\9\3\3\d\6\6 ]] 2024-09-17T23:52:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=dda71afca0613477222ef627c266aa6f 2024-09-17T23:52:53.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-17T23:52:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=85f8d71a7a6755f879fd0e398f4b429c 2024-09-17T23:52:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-17T23:52:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ dda71afca0613477222ef627c266aa6f != \d\d\a\7\1\a\f\c\a\0\6\1\3\4\7\7\2\2\2\e\f\6\2\7\c\2\6\6\a\a\6\f ]] 2024-09-17T23:52:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:52:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 67641 2024-09-17T23:52:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 85f8d71a7a6755f879fd0e398f4b429c != \8\5\f\8\d\7\1\a\7\a\6\7\5\5\f\8\7\9\f\d\0\e\3\9\8\f\4\b\4\2\9\c ]] 2024-09-17T23:52:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:52:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 67643 2024-09-17T23:52:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:52:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 67645 2024-09-17T23:52:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:52:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 67647 2024-09-17T23:52:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:52:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 67649 2024-09-17T23:52:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:52:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 67650 2024-09-17T23:52:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:52:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 67653 2024-09-17T23:52:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:52:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 67654 2024-09-17T23:52:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:52:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 67656 2024-09-17T23:52:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-17T23:52:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:52:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-17T23:52:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-17T23:52:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:52:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:52:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-17T23:52:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:52:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:52:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-17T23:52:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:52:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:52:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-17T23:52:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-17T23:52:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 26114b26-116f-40a8-a2cc-e67f85f7b4c4 2024-09-17T23:52:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:52:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3331276.8,"bytes_per_snapshot":66625536.0,"last_snapshot_bytes":66625536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"} 2024-09-17T23:52:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:47 2024-09-17T23:52:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:52:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:52:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:52:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:52' 2024-09-17T23:52:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:52:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:52:54.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3331276.8,"bytes_per_snapshot":66625536.0,"last_snapshot_bytes":66625536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"} 2024-09-17T23:52:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:52:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:52:54.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:54.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:52:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-17 23:52:47' 2024-09-17T23:52:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:52:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:52:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:47' 2024-09-17T23:52:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary126114b26-116f-40a8-a2cc-e67f85f7b4c4up+replayingreplaying, {"bytes_per_second":3331276.8,"bytes_per_snapshot":66625536.0,"last_snapshot_bytes":66625536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:52image-primary10085fc961-bab1-4386-91c9-86972b75c274up+replayingreplaying, {"bytes_per_second":3235020.8,"bytes_per_snapshot":64700416.0,"last_snapshot_bytes":64700416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:52image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+replayingreplaying, {"bytes_per_second":3068518.4,"bytes_per_snapshot":61370368.0,"last_snapshot_bytes":61370368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:52image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+replayingreplaying, {"bytes_per_second":3503923.2,"bytes_per_snapshot":70078464.0,"last_snapshot_bytes":70078464,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:52image-primary4305debd5-50a6-46f0-baca-0858687046eeup+replayingreplaying, {"bytes_per_second":3497574.4,"bytes_per_snapshot":69951488.0,"last_snapshot_bytes":69951488,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:52image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+replayingreplaying, {"bytes_per_second":3081216.0,"bytes_per_snapshot":61624320.0,"last_snapshot_bytes":61624320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:52image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+replayingreplaying, {"bytes_per_second":3067084.8,"bytes_per_snapshot":61341696.0,"last_snapshot_bytes":61341696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:52image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+replayingreplaying, {"bytes_per_second":3152691.2,"bytes_per_snapshot":63053824.0,"last_snapshot_bytes":63053824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:52image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+replayingreplaying, {"bytes_per_second":3333120.0,"bytes_per_snapshot":66662400.0,"last_snapshot_bytes":66662400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:52image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+replayingreplaying, {"bytes_per_second":3275776.0,"bytes_per_snapshot":65515520.0,"last_snapshot_bytes":65515520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:52' 2024-09-17T23:52:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-17T23:52:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:52:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-17T23:52:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:52:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:52:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:52:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:52:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-17T23:52:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:52:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-17T23:52:54.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-17T23:52:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:52:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-17T23:52:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:52:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:52:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary1 2024-09-17T23:52:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary1 2024-09-17T23:52:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:52:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-17T23:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-17T23:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-17T23:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:52:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-17T23:52:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:52:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:52:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-17T23:52:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-17T23:52:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 7e53aedd-f077-49b3-b8e4-40ec80c224e6 2024-09-17T23:52:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:52:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3068518.4,"bytes_per_snapshot":61370368.0,"last_snapshot_bytes":61370368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"} 2024-09-17T23:52:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:52:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:47 2024-09-17T23:52:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:52:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:52:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:52:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:52:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:57' 2024-09-17T23:52:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:52:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:52:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:52:57.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3068518.4,"bytes_per_snapshot":61370368.0,"last_snapshot_bytes":61370368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"} 2024-09-17T23:52:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:52:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:52:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:52:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:52:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:52:57.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:52:57.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:52:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:52:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:52:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-17 23:52:47' 2024-09-17T23:52:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:52:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:52:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:47' 2024-09-17T23:52:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:52:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:52:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:52:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK942634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10085fc961-bab1-4386-91c9-86972b75c274up+replayingreplaying, {"bytes_per_second":3235020.8,"bytes_per_snapshot":64700416.0,"last_snapshot_bytes":64700416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57image-primary27e53aedd-f077-49b3-b8e4-40ec80c224e6up+replayingreplaying, {"bytes_per_second":3068518.4,"bytes_per_snapshot":61370368.0,"last_snapshot_bytes":61370368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+replayingreplaying, {"bytes_per_second":3503923.2,"bytes_per_snapshot":70078464.0,"last_snapshot_bytes":70078464,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57image-primary4305debd5-50a6-46f0-baca-0858687046eeup+replayingreplaying, {"bytes_per_second":3497574.4,"bytes_per_snapshot":69951488.0,"last_snapshot_bytes":69951488,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+replayingreplaying, {"bytes_per_second":3081216.0,"bytes_per_snapshot":61624320.0,"last_snapshot_bytes":61624320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+replayingreplaying, {"bytes_per_second":3067084.8,"bytes_per_snapshot":61341696.0,"last_snapshot_bytes":61341696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+replayingreplaying, {"bytes_per_second":3152691.2,"bytes_per_snapshot":63053824.0,"last_snapshot_bytes":63053824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+replayingreplaying, {"bytes_per_second":3333120.0,"bytes_per_snapshot":66662400.0,"last_snapshot_bytes":66662400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+replayingreplaying, {"bytes_per_second":3275776.0,"bytes_per_snapshot":65515520.0,"last_snapshot_bytes":65515520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57' 2024-09-17T23:52:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:52:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-17T23:52:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:52:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-17T23:52:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:52:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:52:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:52:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:52:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-17T23:52:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:52:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:52:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-17T23:52:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-17T23:52:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:52:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-17T23:52:59.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:52:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:52:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary2 2024-09-17T23:52:59.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:52:59.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary2 2024-09-17T23:52:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:52:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-17T23:52:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-17T23:52:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:52:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:52:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-17T23:52:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:52:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:52:59.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:52:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:52:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:52:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:52:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:52:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:53:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-17T23:53:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:53:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:53:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:53:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:00.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-17T23:53:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-17T23:53:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: eb4a6a06-f497-481b-9b07-84f8c49bcbd1 2024-09-17T23:53:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:53:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3503923.2,"bytes_per_snapshot":70078464.0,"last_snapshot_bytes":70078464,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"} 2024-09-17T23:53:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:53:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:47 2024-09-17T23:53:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:53:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:53:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:53:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:53:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:57' 2024-09-17T23:53:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:53:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:53:00.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:53:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3503923.2,"bytes_per_snapshot":70078464.0,"last_snapshot_bytes":70078464,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"} 2024-09-17T23:53:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:53:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:53:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:53:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:53:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:53:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:53:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:53:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-17 23:52:47' 2024-09-17T23:53:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:53:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:47' 2024-09-17T23:53:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:53:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK842634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10085fc961-bab1-4386-91c9-86972b75c274up+replayingreplaying, {"bytes_per_second":3235020.8,"bytes_per_snapshot":64700416.0,"last_snapshot_bytes":64700416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57image-primary3eb4a6a06-f497-481b-9b07-84f8c49bcbd1up+replayingreplaying, {"bytes_per_second":3503923.2,"bytes_per_snapshot":70078464.0,"last_snapshot_bytes":70078464,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57image-primary4305debd5-50a6-46f0-baca-0858687046eeup+replayingreplaying, {"bytes_per_second":3497574.4,"bytes_per_snapshot":69951488.0,"last_snapshot_bytes":69951488,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+replayingreplaying, {"bytes_per_second":3081216.0,"bytes_per_snapshot":61624320.0,"last_snapshot_bytes":61624320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+replayingreplaying, {"bytes_per_second":3067084.8,"bytes_per_snapshot":61341696.0,"last_snapshot_bytes":61341696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+replayingreplaying, {"bytes_per_second":3152691.2,"bytes_per_snapshot":63053824.0,"last_snapshot_bytes":63053824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+replayingreplaying, {"bytes_per_second":3333120.0,"bytes_per_snapshot":66662400.0,"last_snapshot_bytes":66662400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+replayingreplaying, {"bytes_per_second":3275776.0,"bytes_per_snapshot":65515520.0,"last_snapshot_bytes":65515520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:52:57' 2024-09-17T23:53:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:53:01.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-17T23:53:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:53:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-17T23:53:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:53:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:53:01.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-17T23:53:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:53:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:53:01.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-17T23:53:01.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-17T23:53:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:53:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-17T23:53:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:53:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:53:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:53:02.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary3 2024-09-17T23:53:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary3 2024-09-17T23:53:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-17T23:53:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-17T23:53:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:53:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-17T23:53:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:53:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:53:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:53:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:53:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-17T23:53:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:53:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:53:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:03.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-17T23:53:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-17T23:53:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 305debd5-50a6-46f0-baca-0858687046ee 2024-09-17T23:53:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:53:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3497574.4,"bytes_per_snapshot":69951488.0,"last_snapshot_bytes":69951488,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"} 2024-09-17T23:53:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:53:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:47 2024-09-17T23:53:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:53:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:53:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:53:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:53:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:53:02' 2024-09-17T23:53:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:53:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:53:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:53:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3497574.4,"bytes_per_snapshot":69951488.0,"last_snapshot_bytes":69951488,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"} 2024-09-17T23:53:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:53:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:53:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:03.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:53:03.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:53:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:53:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:53:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:53:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-17 23:52:47' 2024-09-17T23:53:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:47' 2024-09-17T23:53:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:03.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:53:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK742634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10085fc961-bab1-4386-91c9-86972b75c274up+replayingreplaying, {"bytes_per_second":3235020.8,"bytes_per_snapshot":64700416.0,"last_snapshot_bytes":64700416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:02image-primary4305debd5-50a6-46f0-baca-0858687046eeup+replayingreplaying, {"bytes_per_second":3497574.4,"bytes_per_snapshot":69951488.0,"last_snapshot_bytes":69951488,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:02image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+replayingreplaying, {"bytes_per_second":3081216.0,"bytes_per_snapshot":61624320.0,"last_snapshot_bytes":61624320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:02image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+replayingreplaying, {"bytes_per_second":3067084.8,"bytes_per_snapshot":61341696.0,"last_snapshot_bytes":61341696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:02image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+replayingreplaying, {"bytes_per_second":3152691.2,"bytes_per_snapshot":63053824.0,"last_snapshot_bytes":63053824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:02image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+replayingreplaying, {"bytes_per_second":3333120.0,"bytes_per_snapshot":66662400.0,"last_snapshot_bytes":66662400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:02image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+replayingreplaying, {"bytes_per_second":3275776.0,"bytes_per_snapshot":65515520.0,"last_snapshot_bytes":65515520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:02' 2024-09-17T23:53:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:53:04.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-17T23:53:04.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:53:04.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-17T23:53:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:53:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:53:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-17T23:53:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:53:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-17T23:53:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-17T23:53:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:53:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-17T23:53:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:53:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:53:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary4 2024-09-17T23:53:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary4 2024-09-17T23:53:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-17T23:53:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-17T23:53:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:53:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-17T23:53:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:53:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:53:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:53:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:53:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-17T23:53:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:53:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:53:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-17T23:53:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-17T23:53:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 945834ef-a945-4296-8d1f-7781cb333d6c 2024-09-17T23:53:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:53:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3081216.0,"bytes_per_snapshot":61624320.0,"last_snapshot_bytes":61624320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"} 2024-09-17T23:53:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:53:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:47 2024-09-17T23:53:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:53:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:53:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:53:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:53:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:53:02' 2024-09-17T23:53:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:53:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:53:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:53:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3081216.0,"bytes_per_snapshot":61624320.0,"last_snapshot_bytes":61624320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"} 2024-09-17T23:53:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:53:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:06.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:53:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:53:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:53:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:53:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:53:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:53:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-17 23:52:47' 2024-09-17T23:53:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:47' 2024-09-17T23:53:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:53:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK642634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10085fc961-bab1-4386-91c9-86972b75c274up+replayingreplaying, {"bytes_per_second":3235020.8,"bytes_per_snapshot":64700416.0,"last_snapshot_bytes":64700416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:02image-primary5945834ef-a945-4296-8d1f-7781cb333d6cup+replayingreplaying, {"bytes_per_second":3081216.0,"bytes_per_snapshot":61624320.0,"last_snapshot_bytes":61624320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:02image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+replayingreplaying, {"bytes_per_second":3067084.8,"bytes_per_snapshot":61341696.0,"last_snapshot_bytes":61341696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:02image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+replayingreplaying, {"bytes_per_second":3152691.2,"bytes_per_snapshot":63053824.0,"last_snapshot_bytes":63053824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:02image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+replayingreplaying, {"bytes_per_second":3333120.0,"bytes_per_snapshot":66662400.0,"last_snapshot_bytes":66662400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:02image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+replayingreplaying, {"bytes_per_second":3275776.0,"bytes_per_snapshot":65515520.0,"last_snapshot_bytes":65515520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:02' 2024-09-17T23:53:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:53:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-17T23:53:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:53:07.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-17T23:53:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:53:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:53:07.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-17T23:53:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:53:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-17T23:53:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-17T23:53:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:53:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-17T23:53:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:53:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:53:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary5 2024-09-17T23:53:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary5 2024-09-17T23:53:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-17T23:53:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-17T23:53:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:53:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-17T23:53:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:53:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:53:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:53:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:53:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-17T23:53:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:53:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:53:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:09.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:09.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-17T23:53:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-17T23:53:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: cfe9f926-9ade-41df-8253-26b0f3df4079 2024-09-17T23:53:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:53:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3067084.8,"bytes_per_snapshot":61341696.0,"last_snapshot_bytes":61341696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"} 2024-09-17T23:53:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:53:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:47 2024-09-17T23:53:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:53:07' 2024-09-17T23:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:53:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:53:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:53:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3067084.8,"bytes_per_snapshot":61341696.0,"last_snapshot_bytes":61341696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"} 2024-09-17T23:53:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:53:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:53:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:53:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:53:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:53:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:53:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:53:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-17 23:52:47' 2024-09-17T23:53:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:47' 2024-09-17T23:53:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:53:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK542634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10085fc961-bab1-4386-91c9-86972b75c274up+replayingreplaying, {"bytes_per_second":3235020.8,"bytes_per_snapshot":64700416.0,"last_snapshot_bytes":64700416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:07image-primary6cfe9f926-9ade-41df-8253-26b0f3df4079up+replayingreplaying, {"bytes_per_second":3067084.8,"bytes_per_snapshot":61341696.0,"last_snapshot_bytes":61341696,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:07image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+replayingreplaying, {"bytes_per_second":3152691.2,"bytes_per_snapshot":63053824.0,"last_snapshot_bytes":63053824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:07image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+replayingreplaying, {"bytes_per_second":3333120.0,"bytes_per_snapshot":66662400.0,"last_snapshot_bytes":66662400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:07image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+replayingreplaying, {"bytes_per_second":3275776.0,"bytes_per_snapshot":65515520.0,"last_snapshot_bytes":65515520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:07' 2024-09-17T23:53:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:53:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-17T23:53:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:53:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-17T23:53:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:53:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:53:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-17T23:53:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:53:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-17T23:53:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-17T23:53:10.806 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:53:10.804+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4dd0 2024-09-17T23:53:10.807 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:53:10.804+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ea270 2024-09-17T23:53:10.808 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:53:10.806+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-17T23:53:10.808 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:53:10.806+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-17T23:53:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:53:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-17T23:53:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:53:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:53:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary6 2024-09-17T23:53:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary6 2024-09-17T23:53:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-17T23:53:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-17T23:53:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:53:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-17T23:53:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:53:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:53:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:53:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:53:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-17T23:53:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:53:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:53:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-17T23:53:12.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-17T23:53:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a9650c79-cb43-4f09-baed-431a05d5bac7 2024-09-17T23:53:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:53:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3152691.2,"bytes_per_snapshot":63053824.0,"last_snapshot_bytes":63053824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"} 2024-09-17T23:53:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:53:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:47 2024-09-17T23:53:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:53:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:53:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:53:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:53:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:53:12' 2024-09-17T23:53:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:53:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:53:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:53:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3152691.2,"bytes_per_snapshot":63053824.0,"last_snapshot_bytes":63053824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"} 2024-09-17T23:53:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:53:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:53:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:53:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:53:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:53:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:53:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:53:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-17 23:52:47' 2024-09-17T23:53:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:47' 2024-09-17T23:53:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10085fc961-bab1-4386-91c9-86972b75c274up+replayingreplaying, {"bytes_per_second":3235020.8,"bytes_per_snapshot":64700416.0,"last_snapshot_bytes":64700416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:12image-primary7a9650c79-cb43-4f09-baed-431a05d5bac7up+replayingreplaying, {"bytes_per_second":3152691.2,"bytes_per_snapshot":63053824.0,"last_snapshot_bytes":63053824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:12image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+replayingreplaying, {"bytes_per_second":3333120.0,"bytes_per_snapshot":66662400.0,"last_snapshot_bytes":66662400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:12image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+replayingreplaying, {"bytes_per_second":3275776.0,"bytes_per_snapshot":65515520.0,"last_snapshot_bytes":65515520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:12' 2024-09-17T23:53:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:53:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-17T23:53:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:53:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-17T23:53:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:53:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:53:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-17T23:53:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:53:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-17T23:53:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-17T23:53:14.481 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:53:14.480+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0c737380 2024-09-17T23:53:14.506 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:53:14.504+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0c737380 2024-09-17T23:53:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:53:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-17T23:53:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:53:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:53:14.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary7 2024-09-17T23:53:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary7 2024-09-17T23:53:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-17T23:53:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-17T23:53:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:53:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-17T23:53:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:53:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:53:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:53:14.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:53:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-17T23:53:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:53:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:53:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-17T23:53:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-17T23:53:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ab09e279-9786-4088-9ca5-ffb8f13fe226 2024-09-17T23:53:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:53:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3333120.0,"bytes_per_snapshot":66662400.0,"last_snapshot_bytes":66662400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"} 2024-09-17T23:53:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:53:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:52:47 2024-09-17T23:53:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:53:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:53:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:53:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:53:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:53:12' 2024-09-17T23:53:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:53:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:53:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:53:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3333120.0,"bytes_per_snapshot":66662400.0,"last_snapshot_bytes":66662400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"} 2024-09-17T23:53:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:53:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:53:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:53:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:53:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:53:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:53:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:53:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-17 23:52:47' 2024-09-17T23:53:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:52:47' 2024-09-17T23:53:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:53:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK342634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10085fc961-bab1-4386-91c9-86972b75c274up+replayingreplaying, {"bytes_per_second":3235020.8,"bytes_per_snapshot":64700416.0,"last_snapshot_bytes":64700416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:12image-primary8ab09e279-9786-4088-9ca5-ffb8f13fe226up+replayingreplaying, {"bytes_per_second":3333120.0,"bytes_per_snapshot":66662400.0,"last_snapshot_bytes":66662400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:12image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+replayingreplaying, {"bytes_per_second":3275776.0,"bytes_per_snapshot":65515520.0,"last_snapshot_bytes":65515520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:52:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:12' 2024-09-17T23:53:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:53:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-17T23:53:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:52:47' 2024-09-17T23:53:15.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-17T23:53:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:53:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:53:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-17T23:53:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:53:15.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-17T23:53:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-17T23:53:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:53:17.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-17T23:53:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:53:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:53:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary8 2024-09-17T23:53:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:17.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary8 2024-09-17T23:53:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-17T23:53:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-17T23:53:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:53:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-17T23:53:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:53:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:53:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:53:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:53:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-17T23:53:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:53:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:53:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-17T23:53:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-17T23:53:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921 2024-09-17T23:53:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:53:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65515520.0,"last_snapshot_bytes":65515520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"} 2024-09-17T23:53:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:53:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:53:17 2024-09-17T23:53:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:53:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:53:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:53:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:53:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:53:17' 2024-09-17T23:53:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:53:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:53:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:53:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65515520.0,"last_snapshot_bytes":65515520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"} 2024-09-17T23:53:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:53:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:53:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:53:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:53:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:53:18.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:53:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:53:17' 2024-09-17T23:53:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-17 23:53:17' 2024-09-17T23:53:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:53:17' 2024-09-17T23:53:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:53:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK242634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10085fc961-bab1-4386-91c9-86972b75c274up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64700416.0,"last_snapshot_bytes":64700416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:53:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:17image-primary9c4f6e717-6b65-4fbe-bd1c-5f9d9ddbf921up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65515520.0,"last_snapshot_bytes":65515520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617141,"remote_snapshot_timestamp":1726617141,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:53:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:17' 2024-09-17T23:53:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:53:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-17T23:53:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:53:17' 2024-09-17T23:53:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-17T23:53:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:53:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:53:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-17T23:53:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:53:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-17T23:53:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-17T23:53:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:53:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-17T23:53:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:53:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:53:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary9 2024-09-17T23:53:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:20.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary9 2024-09-17T23:53:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-17T23:53:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-17T23:53:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:53:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-17T23:53:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:53:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:53:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:53:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:53:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-17T23:53:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:53:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:53:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-17T23:53:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-17T23:53:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 085fc961-bab1-4386-91c9-86972b75c274 2024-09-17T23:53:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:53:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64700416.0,"last_snapshot_bytes":64700416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"} 2024-09-17T23:53:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:53:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:53:17 2024-09-17T23:53:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:53:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:53:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:53:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:53:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:53:17' 2024-09-17T23:53:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:53:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:53:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:53:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64700416.0,"last_snapshot_bytes":64700416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"} 2024-09-17T23:53:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:53:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:53:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:53:21.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:53:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:53:21.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:53:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:53:17' 2024-09-17T23:53:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-17 23:53:17' 2024-09-17T23:53:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:53:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:53:17' 2024-09-17T23:53:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:53:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:53:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:53:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10085fc961-bab1-4386-91c9-86972b75c274up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64700416.0,"last_snapshot_bytes":64700416,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617140,"remote_snapshot_timestamp":1726617140,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:53:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:53:17' 2024-09-17T23:53:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:53:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-17T23:53:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:53:17' 2024-09-17T23:53:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-17T23:53:21.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:53:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:53:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:53:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:53:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-17T23:53:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:53:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:21.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:21.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-17T23:53:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-17T23:53:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:53:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-17T23:53:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:53:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:53:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary10 2024-09-17T23:53:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary10 2024-09-17T23:53:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..10} 2024-09-17T23:53:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS=() 2024-09-17T23:53:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS=() 2024-09-17T23:53:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS=() 2024-09-17T23:53:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS=() 2024-09-17T23:53:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ RET=0 2024-09-17T23:53:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-17T23:53:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary1 2024-09-17T23:53:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-17T23:53:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:23.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:53:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:53:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-17T23:53:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:53:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:53:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:53:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:53:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-17T23:53:23.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:53:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:53:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-17T23:53:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:53:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-17T23:53:23.974 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:53:23.972+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-17T23:53:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:53:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-17T23:53:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary1': 2024-09-17T23:53:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:53:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:53:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:53:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1e445daeb0c6 2024-09-17T23:53:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1e445daeb0c6 2024-09-17T23:53:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:53:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:53:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:53:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:53:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:53:23 2024 2024-09-17T23:53:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:53:23 2024 2024-09-17T23:53:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:53:23 2024 2024-09-17T23:53:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:53:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:53:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 24918fa8-5a55-499c-98f9-3d026778b128 2024-09-17T23:53:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:53:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:53:24.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:53:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-17T23:53:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd0 2024-09-17T23:53:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:53:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-17T23:53:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:53:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:53:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:53:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 01b3a90e-4036-4f65-8870-26faa0c0f1d3 2024-09-17T23:53:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:53:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:53:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:53:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:53:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:53:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:53:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary1 2024-09-17T23:53:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-17T23:53:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:53:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary1 2024-09-17T23:53:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:53:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-17T23:53:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:53:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:53:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:53:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-17T23:53:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary1 2024-09-17T23:53:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary2 2024-09-17T23:53:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:53:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-17T23:53:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:53:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary1 2024-09-17T23:53:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-17T23:53:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:53:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:53:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-17T23:53:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:53:24.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:53:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:53:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:53:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-17T23:53:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:53:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:53:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-17T23:53:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:53:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-17T23:53:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:53:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-17T23:53:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:53:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 995 2024-09-17T23:53:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-17T23:53:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary2': 2024-09-17T23:53:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:53:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:53:25.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:53:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1e54ff033960 2024-09-17T23:53:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1e54ff033960 2024-09-17T23:53:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:53:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:53:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:53:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:53:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:53:24 2024 2024-09-17T23:53:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:53:24 2024 2024-09-17T23:53:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:53:24 2024 2024-09-17T23:53:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:53:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:53:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 0b7bb811-ff64-4199-9949-17659e6a10ed 2024-09-17T23:53:25.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:53:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:53:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:53:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-17T23:53:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd1 2024-09-17T23:53:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:53:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-17T23:53:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:53:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:53:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:53:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: f24122a2-a8a1-45db-a73f-58aee60b098b 2024-09-17T23:53:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:53:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:53:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:53:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:53:25.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:53:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:53:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary2 2024-09-17T23:53:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-17T23:53:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:53:25.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary2 2024-09-17T23:53:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:53:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-17T23:53:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:53:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:53:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-17T23:53:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:53:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary3 2024-09-17T23:53:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-17T23:53:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary2 2024-09-17T23:53:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:53:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:53:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:53:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:53:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-17T23:53:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary2 2024-09-17T23:53:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:53:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:53:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:53:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:53:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-17T23:53:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-17T23:53:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:53:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:53:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-17T23:53:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:53:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-17T23:53:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:53:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-17T23:53:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:53:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 997 2024-09-17T23:53:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-17T23:53:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary3': 2024-09-17T23:53:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:53:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:53:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:53:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1e6377146efe 2024-09-17T23:53:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1e6377146efe 2024-09-17T23:53:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:53:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:53:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:53:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:53:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:53:25 2024 2024-09-17T23:53:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:53:25 2024 2024-09-17T23:53:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:53:25 2024 2024-09-17T23:53:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:53:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:53:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: d50bf219-7b94-4aed-ac1f-03c36a7a6386 2024-09-17T23:53:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:53:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:53:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:53:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-17T23:53:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd2 2024-09-17T23:53:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:53:26.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-17T23:53:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:53:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:53:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:53:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: c8a91a80-d361-46b3-b3e2-d36f5ea50ea9 2024-09-17T23:53:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:53:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:53:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:53:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:53:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:53:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:53:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary3 2024-09-17T23:53:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-17T23:53:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:53:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary3 2024-09-17T23:53:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:53:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-17T23:53:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:53:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:53:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:53:26.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-17T23:53:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary4 2024-09-17T23:53:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary3 2024-09-17T23:53:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-17T23:53:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:53:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:53:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:53:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary3 2024-09-17T23:53:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-17T23:53:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-17T23:53:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:53:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:53:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:53:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-17T23:53:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:53:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:53:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-17T23:53:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-17T23:53:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:53:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-17T23:53:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:53:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 999 2024-09-17T23:53:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-17T23:53:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary4': 2024-09-17T23:53:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:53:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:53:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:53:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1e72835cfb67 2024-09-17T23:53:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1e72835cfb67 2024-09-17T23:53:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:53:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:53:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:53:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:53:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:53:26 2024 2024-09-17T23:53:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:53:26 2024 2024-09-17T23:53:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:53:26 2024 2024-09-17T23:53:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:53:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:53:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 07a8d24c-e200-4a35-a6cb-a1ab753f27bf 2024-09-17T23:53:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:53:27.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:53:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:53:27.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-17T23:53:27.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd3 2024-09-17T23:53:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:53:27.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-17T23:53:27.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:53:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:53:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:53:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 87556d5d-6512-4012-aa1a-d7dcce5dcc9f 2024-09-17T23:53:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:53:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:53:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:53:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:53:27.672 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-17T23:53:27.669+0000 7f6acfc5f640 -1 reset not still connected to 0x555c09ea0d00 2024-09-17T23:53:27.672 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-17T23:53:27.669+0000 7f6acfc5f640 -1 reset not still connected to 0x555c09ea1d40 2024-09-17T23:53:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:53:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:53:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:27.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary4 2024-09-17T23:53:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-17T23:53:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:53:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary4 2024-09-17T23:53:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:53:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-17T23:53:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:53:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:53:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary4 2024-09-17T23:53:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:53:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-17T23:53:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary5 2024-09-17T23:53:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-17T23:53:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary4 2024-09-17T23:53:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-17T23:53:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:53:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-17T23:53:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:53:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:53:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:53:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-17T23:53:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:53:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-17T23:53:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:53:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:53:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-17T23:53:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-17T23:53:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:53:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1001 2024-09-17T23:53:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:53:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:53:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:53:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-17T23:53:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary5': 2024-09-17T23:53:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:53:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:53:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:53:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1e81d1985874 2024-09-17T23:53:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1e81d1985874 2024-09-17T23:53:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:53:28.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:53:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:53:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:53:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:53:28 2024 2024-09-17T23:53:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:53:28 2024 2024-09-17T23:53:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:53:28 2024 2024-09-17T23:53:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:53:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:53:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8171daf5-079e-42db-ab46-78768904f13b 2024-09-17T23:53:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:53:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:53:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:53:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-17T23:53:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd4 2024-09-17T23:53:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:53:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-17T23:53:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:53:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:53:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:53:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: af84a692-d915-47c2-a5c9-631b0d6d6223 2024-09-17T23:53:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:53:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:53:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:53:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:53:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:53:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:53:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary5 2024-09-17T23:53:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-17T23:53:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:53:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary5 2024-09-17T23:53:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:53:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-17T23:53:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:53:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-17T23:53:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary6 2024-09-17T23:53:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-17T23:53:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:53:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:53:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:53:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-17T23:53:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:53:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:53:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:53:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-17T23:53:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary5 2024-09-17T23:53:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary5 2024-09-17T23:53:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-17T23:53:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:53:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:53:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-17T23:53:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-17T23:53:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:53:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-17T23:53:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:53:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1004 2024-09-17T23:53:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-17T23:53:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:53:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:53:29.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:53:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary6': 2024-09-17T23:53:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:53:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:53:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:53:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1e93e936aa1 2024-09-17T23:53:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1e93e936aa1 2024-09-17T23:53:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:53:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:53:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:53:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:53:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:53:28 2024 2024-09-17T23:53:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:53:28 2024 2024-09-17T23:53:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:53:28 2024 2024-09-17T23:53:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:53:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:53:29.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 1b62056c-37af-4377-8ef5-6fd1b5c24a62 2024-09-17T23:53:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:53:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:53:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:53:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-17T23:53:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd5 2024-09-17T23:53:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:53:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-17T23:53:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:53:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:53:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:53:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 87c458d6-f0a8-4059-a278-02e4232ec138 2024-09-17T23:53:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:53:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:53:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:53:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:53:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:53:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:53:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary6 2024-09-17T23:53:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-17T23:53:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:53:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary6 2024-09-17T23:53:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:53:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-17T23:53:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:53:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:29.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:53:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary6 2024-09-17T23:53:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-17T23:53:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary7 2024-09-17T23:53:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-17T23:53:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:53:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary6 2024-09-17T23:53:29.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-17T23:53:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:53:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-17T23:53:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:29.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:53:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:53:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:53:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-17T23:53:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:53:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:53:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-17T23:53:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-17T23:53:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:53:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-17T23:53:30.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1003 2024-09-17T23:53:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:53:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:53:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:53:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:53:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-17T23:53:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1007 2024-09-17T23:53:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary7': 2024-09-17T23:53:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:53:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:53:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:53:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1ea2d85357f9 2024-09-17T23:53:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1ea2d85357f9 2024-09-17T23:53:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:53:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:53:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:53:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:53:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:53:29 2024 2024-09-17T23:53:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:53:29 2024 2024-09-17T23:53:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:53:29 2024 2024-09-17T23:53:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:53:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:53:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 04e5f6d7-d750-4252-b6b3-7e6d97d62e06 2024-09-17T23:53:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:53:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:53:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:53:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-17T23:53:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd6 2024-09-17T23:53:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:53:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-17T23:53:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:53:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:53:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:53:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 4632688c-edd3-4019-8700-6d9af1bd4b7f 2024-09-17T23:53:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:53:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:53:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:53:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:53:30.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:53:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:53:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary7 2024-09-17T23:53:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-17T23:53:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:53:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary7 2024-09-17T23:53:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:53:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-17T23:53:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:53:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:53:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary7 2024-09-17T23:53:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-17T23:53:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary8 2024-09-17T23:53:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-17T23:53:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:53:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:53:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary7 2024-09-17T23:53:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-17T23:53:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-17T23:53:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:30.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:53:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:53:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:53:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:30.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-17T23:53:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:53:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:53:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-17T23:53:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-17T23:53:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:53:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-17T23:53:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1006 2024-09-17T23:53:31.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:53:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:53:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:31.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:53:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1010 2024-09-17T23:53:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:31.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-17T23:53:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary8': 2024-09-17T23:53:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:53:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:53:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:53:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1eb5c5d9a404 2024-09-17T23:53:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1eb5c5d9a404 2024-09-17T23:53:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:53:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:53:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:53:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:53:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:53:30 2024 2024-09-17T23:53:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:53:30 2024 2024-09-17T23:53:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:53:30 2024 2024-09-17T23:53:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:53:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:53:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9a4e684a-d528-43f1-9e0b-f06c8eaea047 2024-09-17T23:53:31.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:53:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:53:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:53:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-17T23:53:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd7 2024-09-17T23:53:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:53:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-17T23:53:31.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:53:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:53:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:53:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 4e10fa63-f572-4de2-83d2-8c1145014fe0 2024-09-17T23:53:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:53:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:53:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:53:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:53:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:53:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:53:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary8 2024-09-17T23:53:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-17T23:53:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:53:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary8 2024-09-17T23:53:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:53:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-17T23:53:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:53:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:53:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary8 2024-09-17T23:53:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-17T23:53:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary9 2024-09-17T23:53:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:53:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-17T23:53:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary8 2024-09-17T23:53:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-17T23:53:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:53:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-17T23:53:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:53:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:53:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:53:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-17T23:53:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:53:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:53:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-17T23:53:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-17T23:53:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:53:32.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-17T23:53:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1009 2024-09-17T23:53:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:53:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:53:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:53:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1013 2024-09-17T23:53:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-17T23:53:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary9': 2024-09-17T23:53:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:53:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:53:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:53:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1ec66c0e319a 2024-09-17T23:53:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1ec66c0e319a 2024-09-17T23:53:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:53:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:53:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:53:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:53:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:53:32 2024 2024-09-17T23:53:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:53:32 2024 2024-09-17T23:53:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:53:32 2024 2024-09-17T23:53:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:53:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:53:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 7e11cb55-df54-4319-b4ee-da4847b7d252 2024-09-17T23:53:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:53:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:53:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:53:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-17T23:53:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd8 2024-09-17T23:53:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:53:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-17T23:53:32.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 201f0313-e63b-4e95-a317-be807757e06f 2024-09-17T23:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:53:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:53:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:53:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:53:32.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary9 2024-09-17T23:53:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-17T23:53:32.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:53:32.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary9 2024-09-17T23:53:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:53:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-17T23:53:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:53:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:53:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:53:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-17T23:53:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary9 2024-09-17T23:53:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:32.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary10 2024-09-17T23:53:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-17T23:53:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:53:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary9 2024-09-17T23:53:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-17T23:53:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-17T23:53:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-17T23:53:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-17T23:53:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-17T23:53:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-17T23:53:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-17T23:53:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-17T23:53:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-17T23:53:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-17T23:53:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:53:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-17T23:53:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-17T23:53:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-17T23:53:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:53:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-17T23:53:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1012 2024-09-17T23:53:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:53:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:53:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:53:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1016 2024-09-17T23:53:33.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:53:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:53:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-17T23:53:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-17T23:53:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary10': 2024-09-17T23:53:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-17T23:53:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-17T23:53:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-17T23:53:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 1ed82bdcce15 2024-09-17T23:53:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.1ed82bdcce15 2024-09-17T23:53:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-17T23:53:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-17T23:53:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-17T23:53:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-17T23:53:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Tue Sep 17 23:53:33 2024 2024-09-17T23:53:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Tue Sep 17 23:53:33 2024 2024-09-17T23:53:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Tue Sep 17 23:53:33 2024 2024-09-17T23:53:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-17T23:53:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-17T23:53:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 801c5de6-b3f3-4fca-ba00-b00b417252d2 2024-09-17T23:53:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-17T23:53:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:53:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:53:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-17T23:53:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd9 2024-09-17T23:53:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-17T23:53:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-17T23:53:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-17T23:53:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-17T23:53:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-17T23:53:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: c33f4865-5e6b-46fe-abb6-59ea7fc69f74 2024-09-17T23:53:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-17T23:53:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-17T23:53:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-17T23:53:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-17T23:53:33.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-17T23:53:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-17T23:53:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-17T23:53:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary10 2024-09-17T23:53:33.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-17T23:53:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-17T23:53:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary10 2024-09-17T23:53:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-17T23:53:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-17T23:53:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-17T23:53:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:53:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary10 2024-09-17T23:53:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 71206 2024-09-17T23:53:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:53:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary10 2024-09-17T23:53:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-17T23:53:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:53:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-17T23:53:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-17T23:53:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1020 2024-09-17T23:53:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:53:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:53:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:53:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:34.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:53:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:53:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1015 2024-09-17T23:53:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1019 2024-09-17T23:53:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:53:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:53:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:53:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:53:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:53:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:35.946 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:53:35.943+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1d99ed00 2024-09-17T23:53:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1018 2024-09-17T23:53:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1021 2024-09-17T23:53:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:53:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:53:36.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:53:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:53:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:53:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:53:37.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1022 2024-09-17T23:53:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1023 2024-09-17T23:53:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:53:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:53:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1024 2024-09-17T23:53:38.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1026 2024-09-17T23:53:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1025 2024-09-17T23:53:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:53:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:53:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:53:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1027 2024-09-17T23:53:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1028 2024-09-17T23:53:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:53:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:53:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:53:39.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:53:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:53:40.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1029 2024-09-17T23:53:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1030 2024-09-17T23:53:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:53:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:53:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:53:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:53:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:53:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:53:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:53:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:53:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:53:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:53:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:53:41.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1033 2024-09-17T23:53:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:53:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:53:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:53:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:53:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1032 2024-09-17T23:53:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1031 2024-09-17T23:53:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:42.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1034 2024-09-17T23:53:42.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:42.905 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:53:42.903+0000 7f9bfab49640 -1 reset not still connected to 0x55cd15d401a0 2024-09-17T23:53:42.905 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:53:42.903+0000 7f9bfab49640 -1 reset not still connected to 0x55cd20ad6dd0 2024-09-17T23:53:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:53:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:53:43.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:53:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:53:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:53:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1036 2024-09-17T23:53:43.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1035 2024-09-17T23:53:43.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1037 2024-09-17T23:53:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1038 2024-09-17T23:53:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:53:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:44.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:44.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:53:44.968 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:53:44.966+0000 7f9bfab49640 -1 reset not still connected to 0x55cd20ad6dd0 2024-09-17T23:53:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1040 2024-09-17T23:53:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1039 2024-09-17T23:53:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:53:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:53:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:53:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:53:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:53:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:53:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:53:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:53:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1041 2024-09-17T23:53:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1043 2024-09-17T23:53:46.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:53:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:53:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:53:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:53:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1042 2024-09-17T23:53:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1044 2024-09-17T23:53:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:53:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:53:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:53:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:53:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1046 2024-09-17T23:53:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:53:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:53:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:53:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:53:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:48.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:53:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1045 2024-09-17T23:53:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1048 2024-09-17T23:53:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1047 2024-09-17T23:53:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:53:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:53:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:53:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:53:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:53:50.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1050 2024-09-17T23:53:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:53:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:53:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:53:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:53:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:53:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1052 2024-09-17T23:53:51.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1049 2024-09-17T23:53:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:53:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:53:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:53:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:51.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:53:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:53:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:53:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1054 2024-09-17T23:53:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1051 2024-09-17T23:53:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:53:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:53:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:52.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1055 2024-09-17T23:53:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1053 2024-09-17T23:53:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1056 2024-09-17T23:53:53.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:53:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:53.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:53:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:53.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1058 2024-09-17T23:53:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1057 2024-09-17T23:53:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:53:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:53:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:53:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:53:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:53:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:53:55.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1059 2024-09-17T23:53:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:53:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:53:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1061 2024-09-17T23:53:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:53:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:53:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:53:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1060 2024-09-17T23:53:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1062 2024-09-17T23:53:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:53:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:53:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:53:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:53:56.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:53:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:53:56.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:53:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:53:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:53:57.366 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:53:57.364+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4dd0 2024-09-17T23:53:57.366 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:53:57.364+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb5790 2024-09-17T23:53:57.366 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:53:57.364+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ea270 2024-09-17T23:53:57.367 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:53:57.365+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-17T23:53:57.367 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:53:57.365+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-17T23:53:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:53:57.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:53:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:53:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:53:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:53:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1065 2024-09-17T23:53:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1064 2024-09-17T23:53:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1066 2024-09-17T23:53:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1063 2024-09-17T23:53:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:53:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:53:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:53:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:53:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:53:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:53:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1067 2024-09-17T23:53:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1068 2024-09-17T23:53:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:53:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:53:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:53:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:53:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:53:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:53:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:53:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:53:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:53:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:53:59.676 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:53:59.674+0000 7fc275b74640 -1 reset not still connected to 0x559cc3ed0820 2024-09-17T23:53:59.676 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:53:59.674+0000 7fc275b74640 -1 reset not still connected to 0x559ccd2cef70 2024-09-17T23:54:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1069 2024-09-17T23:54:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1070 2024-09-17T23:54:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:00.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:00.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:00.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:54:00.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:54:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:54:01.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1071 2024-09-17T23:54:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:54:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:54:01.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:54:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:54:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:54:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:54:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:54:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:54:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:54:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:54:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:54:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:54:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:54:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:54:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:54:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1072 2024-09-17T23:54:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1076 2024-09-17T23:54:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1073 2024-09-17T23:54:03.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1075 2024-09-17T23:54:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1074 2024-09-17T23:54:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:54:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:54:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:54:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:54:03.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:54:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:54:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1077 2024-09-17T23:54:04.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1078 2024-09-17T23:54:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:54:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:54:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:54:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1079 2024-09-17T23:54:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1080 2024-09-17T23:54:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:05.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:54:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:54:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:54:05.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:05.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:54:05.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:05.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:05.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:54:05.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:54:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1081 2024-09-17T23:54:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1082 2024-09-17T23:54:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:54:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:54:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:54:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:54:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:54:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:54:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:54:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:54:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:54:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1083 2024-09-17T23:54:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:54:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:54:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:54:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:54:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:54:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:54:07.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1084 2024-09-17T23:54:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1085 2024-09-17T23:54:08.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1086 2024-09-17T23:54:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:54:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:54:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:54:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:54:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:54:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:54:08.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1087 2024-09-17T23:54:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1088 2024-09-17T23:54:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:54:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:54:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:54:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:54:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:54:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:54:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:54:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:54:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:54:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1089 2024-09-17T23:54:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1090 2024-09-17T23:54:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:54:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:54:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:54:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1092 2024-09-17T23:54:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1091 2024-09-17T23:54:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:54:11.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:11.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:11.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:54:11.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:54:11.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:11.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:54:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:54:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:54:11.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1094 2024-09-17T23:54:12.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1093 2024-09-17T23:54:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:54:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:54:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:54:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:54:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:54:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:54:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:12.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:13.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1096 2024-09-17T23:54:13.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1095 2024-09-17T23:54:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:54:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:13.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:54:13.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:54:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:54:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:54:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:54:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1097 2024-09-17T23:54:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1098 2024-09-17T23:54:14.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:54:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:54:14.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:54:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:54:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:54:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:54:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1099 2024-09-17T23:54:15.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1100 2024-09-17T23:54:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:15.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:54:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:54:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:54:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:54:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:54:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:54:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1101 2024-09-17T23:54:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:54:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:54:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:54:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:54:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:54:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:54:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:54:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:54:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:54:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:54:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:54:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:54:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1104 2024-09-17T23:54:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1102 2024-09-17T23:54:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1103 2024-09-17T23:54:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1105 2024-09-17T23:54:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1106 2024-09-17T23:54:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:54:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:54:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:54:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:54:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:54:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:54:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1108 2024-09-17T23:54:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1107 2024-09-17T23:54:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:54:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:54:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:54:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1110 2024-09-17T23:54:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1109 2024-09-17T23:54:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:54:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:54:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:54:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:20.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:54:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:54:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:54:20.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:21.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1111 2024-09-17T23:54:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:54:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:54:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:54:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:54:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:21.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:54:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:54:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:54:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:54:21.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:54:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1113 2024-09-17T23:54:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:54:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:54:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:54:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:54:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:54:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:54:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1115 2024-09-17T23:54:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1117 2024-09-17T23:54:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1116 2024-09-17T23:54:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1114 2024-09-17T23:54:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:54:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:54:23.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:54:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:54:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:54:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:54:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:23.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:24.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1119 2024-09-17T23:54:24.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1118 2024-09-17T23:54:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:54:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:54:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:54:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:54:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:54:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:54:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1120 2024-09-17T23:54:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1121 2024-09-17T23:54:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:54:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:54:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:54:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1122 2024-09-17T23:54:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1123 2024-09-17T23:54:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:54:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:54:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:54:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:54:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:54:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:54:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:54:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:54:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:54:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:54:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:54:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:54:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:54:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:54:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:54:28.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1126 2024-09-17T23:54:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1124 2024-09-17T23:54:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1125 2024-09-17T23:54:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:54:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:54:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:54:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:54:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:54:28.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:54:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1127 2024-09-17T23:54:29.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1129 2024-09-17T23:54:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1128 2024-09-17T23:54:29.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:54:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:54:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:54:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:54:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:54:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:54:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1130 2024-09-17T23:54:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1131 2024-09-17T23:54:30.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:30.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1132 2024-09-17T23:54:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1133 2024-09-17T23:54:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:54:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:54:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:54:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:54:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:54:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:54:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:54:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:54:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:54:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:54:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:54:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:54:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:54:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:54:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:54:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:54:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:54:32.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:54:32.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1136 2024-09-17T23:54:32.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1135 2024-09-17T23:54:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1134 2024-09-17T23:54:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:54:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:54:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:54:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:54:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:54:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:54:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1137 2024-09-17T23:54:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1138 2024-09-17T23:54:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1139 2024-09-17T23:54:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:54:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:54:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:54:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:54:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:54:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:54:35.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1140 2024-09-17T23:54:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1141 2024-09-17T23:54:35.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:54:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:54:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:54:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1143 2024-09-17T23:54:36.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1142 2024-09-17T23:54:36.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:54:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:54:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:54:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:54:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:54:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:54:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:54:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:54:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:54:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:54:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:54:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:54:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:54:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:54:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:54:37.739 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:54:37.737+0000 7fc275b74640 -1 reset not still connected to 0x559cc3ed0820 2024-09-17T23:54:37.741 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:54:37.738+0000 7fc275b74640 -1 reset not still connected to 0x559ccd2cef70 2024-09-17T23:54:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1145 2024-09-17T23:54:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:54:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:54:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:54:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1144 2024-09-17T23:54:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1146 2024-09-17T23:54:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1147 2024-09-17T23:54:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:54:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:54:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:54:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1148 2024-09-17T23:54:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:39.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:54:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:54:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:54:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:54:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:54:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:54:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1149 2024-09-17T23:54:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1151 2024-09-17T23:54:40.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1150 2024-09-17T23:54:41.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1153 2024-09-17T23:54:41.463 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:54:41.462+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0b291450 2024-09-17T23:54:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:54:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:54:41.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:54:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:54:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:54:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:54:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:54:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:54:41.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:54:41.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:54:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:54:41.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:54:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1152 2024-09-17T23:54:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:54:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:54:42.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:54:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1155 2024-09-17T23:54:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:43.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1157 2024-09-17T23:54:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1154 2024-09-17T23:54:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:54:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:54:43.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:54:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:54:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:54:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:54:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1158 2024-09-17T23:54:44.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1156 2024-09-17T23:54:44.447 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:54:44.444+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0dfd5ee0 2024-09-17T23:54:44.447 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:54:44.444+0000 7f9bfab49640 -1 reset not still connected to 0x55cd15d401a0 2024-09-17T23:54:44.447 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:54:44.445+0000 7f9bfab49640 -1 reset not still connected to 0x55cd20ad6dd0 2024-09-17T23:54:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1159 2024-09-17T23:54:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:54:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:54:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:54:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:54:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:54:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:54:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1160 2024-09-17T23:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:54:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:54:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1162 2024-09-17T23:54:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:54:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:54:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:54:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1161 2024-09-17T23:54:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1163 2024-09-17T23:54:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:54:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:54:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:54:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:54:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:54:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:54:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:54:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:54:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:54:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:54:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:54:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:54:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1164 2024-09-17T23:54:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:54:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:54:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:54:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:48.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1165 2024-09-17T23:54:48.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1166 2024-09-17T23:54:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:54:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:54:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:54:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:54:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:54:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:54:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1167 2024-09-17T23:54:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1168 2024-09-17T23:54:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:54:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:54:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:54:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:49.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:54:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:54:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:54:50.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1171 2024-09-17T23:54:50.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1169 2024-09-17T23:54:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:54:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:54:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:54:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1173 2024-09-17T23:54:51.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1170 2024-09-17T23:54:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1172 2024-09-17T23:54:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:54:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:54:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:54:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:54:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:54:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:54:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:51.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:54:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:54:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:54:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:54:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:54:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1174 2024-09-17T23:54:52.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:53.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1175 2024-09-17T23:54:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1176 2024-09-17T23:54:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1177 2024-09-17T23:54:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:54:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:53.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:53.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:54:53.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:54:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:54:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:54:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:54:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1178 2024-09-17T23:54:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1180 2024-09-17T23:54:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:54:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:54:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:54:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:54:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:54:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:54:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:54:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:54:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:54:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1179 2024-09-17T23:54:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:54:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:54:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:54:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1182 2024-09-17T23:54:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1183 2024-09-17T23:54:56.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1181 2024-09-17T23:54:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:54:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:54:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:54:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:54:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:54:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:54:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:54:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:54:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:54:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:54:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:54:57.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:54:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:54:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:54:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:54:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1184 2024-09-17T23:54:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1185 2024-09-17T23:54:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1187 2024-09-17T23:54:58.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1186 2024-09-17T23:54:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:54:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:54:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:54:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:58.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:54:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:54:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:54:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:58.581 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:54:58.579+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0dfd5ee0 2024-09-17T23:54:58.583 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:54:58.580+0000 7f9bfab49640 -1 reset not still connected to 0x55cd15d401a0 2024-09-17T23:54:58.583 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:54:58.581+0000 7f9bfab49640 -1 reset not still connected to 0x55cd20ad6dd0 2024-09-17T23:54:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1189 2024-09-17T23:54:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1188 2024-09-17T23:54:59.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:54:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:54:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:54:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:54:59.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:54:59.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:54:59.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:54:59.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:54:59.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:54:59.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:55:00.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1190 2024-09-17T23:55:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1191 2024-09-17T23:55:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:55:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:55:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:55:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1193 2024-09-17T23:55:01.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1192 2024-09-17T23:55:01.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:55:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:55:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:55:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:55:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:55:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:55:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:55:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:55:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:55:02.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1194 2024-09-17T23:55:02.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:55:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:55:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:55:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:55:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:55:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:55:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1197 2024-09-17T23:55:03.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1195 2024-09-17T23:55:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1196 2024-09-17T23:55:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:55:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:55:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:55:03.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:55:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:55:03.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:55:03.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1199 2024-09-17T23:55:04.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1198 2024-09-17T23:55:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:55:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:55:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:55:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:55:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:55:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:55:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:05.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1200 2024-09-17T23:55:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:55:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:55:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:55:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1201 2024-09-17T23:55:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:05.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:06.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1202 2024-09-17T23:55:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:55:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:55:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:55:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:55:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:55:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:55:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:55:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:55:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:55:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1204 2024-09-17T23:55:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1203 2024-09-17T23:55:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:55:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:55:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:55:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:55:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:07.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:55:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:55:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:07.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1206 2024-09-17T23:55:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1205 2024-09-17T23:55:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1207 2024-09-17T23:55:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:55:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:55:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:55:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:55:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:55:08.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:55:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:09.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1208 2024-09-17T23:55:09.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1209 2024-09-17T23:55:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:55:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:55:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:55:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1210 2024-09-17T23:55:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1211 2024-09-17T23:55:10.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:55:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:55:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:55:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:55:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:55:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:55:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1212 2024-09-17T23:55:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:55:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:55:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:55:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:55:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:55:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:55:11.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:55:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:55:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:55:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1213 2024-09-17T23:55:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:55:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:55:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:55:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:55:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:55:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:55:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1214 2024-09-17T23:55:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1215 2024-09-17T23:55:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1217 2024-09-17T23:55:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1216 2024-09-17T23:55:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:55:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:55:13.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:55:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:55:13.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:55:13.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:55:13.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1218 2024-09-17T23:55:14.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:14.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:55:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:55:14.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:55:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:55:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:55:14.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:55:14.693 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:55:14.691+0000 7fc275b74640 -1 reset not still connected to 0x559cc3ed0820 2024-09-17T23:55:14.695 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:55:14.693+0000 7fc275b74640 -1 reset not still connected to 0x559cc9206f70 2024-09-17T23:55:14.695 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:55:14.693+0000 7fc275b74640 -1 reset not still connected to 0x559ccd2cef70 2024-09-17T23:55:15.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1220 2024-09-17T23:55:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1219 2024-09-17T23:55:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:55:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:55:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:55:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:15.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1222 2024-09-17T23:55:16.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1221 2024-09-17T23:55:16.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1223 2024-09-17T23:55:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:55:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:55:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:55:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:55:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:55:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:55:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:55:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:16.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:55:16.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:55:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:55:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:55:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:55:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1224 2024-09-17T23:55:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1226 2024-09-17T23:55:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1225 2024-09-17T23:55:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1227 2024-09-17T23:55:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:55:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:55:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:55:18.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:55:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:55:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:55:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1228 2024-09-17T23:55:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1230 2024-09-17T23:55:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:55:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:55:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:55:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:55:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:55:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:55:19.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:19.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:55:19.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:19.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:19.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:55:19.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:55:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1229 2024-09-17T23:55:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:20.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1232 2024-09-17T23:55:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:55:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:55:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:55:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1231 2024-09-17T23:55:21.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1233 2024-09-17T23:55:21.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:21.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:55:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:55:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:55:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:55:21.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:55:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:55:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:55:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:55:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1234 2024-09-17T23:55:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:55:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:55:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:55:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:55:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:55:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:55:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1236 2024-09-17T23:55:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1235 2024-09-17T23:55:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:55:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:55:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:55:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:55:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:55:23.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:55:23.793 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:55:23.791+0000 7fc275b74640 -1 reset not still connected to 0x559cbebc1520 2024-09-17T23:55:23.795 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:55:23.792+0000 7fc275b74640 -1 reset not still connected to 0x559cbee89a00 2024-09-17T23:55:23.795 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:55:23.792+0000 7fc275b74640 -1 reset not still connected to 0x559cbf1f44e0 2024-09-17T23:55:23.795 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:55:23.793+0000 7fc275b74640 -1 reset not still connected to 0x559cbfceb520 2024-09-17T23:55:23.795 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:55:23.793+0000 7fc275b74640 -1 reset not still connected to 0x559cc3ed0820 2024-09-17T23:55:23.795 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:55:23.793+0000 7fc275b74640 -1 reset not still connected to 0x559cc9206f70 2024-09-17T23:55:23.795 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:55:23.793+0000 7fc275b74640 -1 reset not still connected to 0x559ccd2cef70 2024-09-17T23:55:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1239 2024-09-17T23:55:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1238 2024-09-17T23:55:24.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1237 2024-09-17T23:55:24.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:24.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:55:24.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:24.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:24.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:55:24.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:55:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:55:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:55:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:55:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1240 2024-09-17T23:55:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:55:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:55:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:55:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1241 2024-09-17T23:55:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:26.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1242 2024-09-17T23:55:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:55:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:55:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:55:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:55:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:55:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:55:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1244 2024-09-17T23:55:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1243 2024-09-17T23:55:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:55:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:55:27.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:55:27.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:55:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:55:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:55:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:55:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:55:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:55:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1245 2024-09-17T23:55:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1246 2024-09-17T23:55:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:55:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:55:28.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:55:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:55:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:55:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:55:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1249 2024-09-17T23:55:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1247 2024-09-17T23:55:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1248 2024-09-17T23:55:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:55:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:55:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:55:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1251 2024-09-17T23:55:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1250 2024-09-17T23:55:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:55:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:55:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:55:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:55:30.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:55:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:55:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1252 2024-09-17T23:55:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:55:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:55:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:55:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:55:31.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:55:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:55:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1254 2024-09-17T23:55:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:55:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:55:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:55:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1253 2024-09-17T23:55:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:55:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:55:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:55:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:55:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:55:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:55:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1255 2024-09-17T23:55:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1257 2024-09-17T23:55:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1256 2024-09-17T23:55:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:55:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:55:33.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:55:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:55:33.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:55:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:55:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:34.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1258 2024-09-17T23:55:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1259 2024-09-17T23:55:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:55:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:55:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:55:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:55:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:55:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:55:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1260 2024-09-17T23:55:35.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1261 2024-09-17T23:55:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:55:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:55:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:55:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1262 2024-09-17T23:55:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1263 2024-09-17T23:55:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:55:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:55:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:55:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:55:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:55:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:55:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:55:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:55:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:55:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:55:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:55:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:55:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:55:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:55:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:55:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1264 2024-09-17T23:55:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1265 2024-09-17T23:55:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1266 2024-09-17T23:55:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1267 2024-09-17T23:55:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:55:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:55:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:55:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:55:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:55:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:55:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1269 2024-09-17T23:55:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1268 2024-09-17T23:55:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:55:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:55:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:55:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:55:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:55:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:55:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1271 2024-09-17T23:55:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1270 2024-09-17T23:55:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:55:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:55:40.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:55:40.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:40.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1273 2024-09-17T23:55:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1272 2024-09-17T23:55:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:55:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:55:41.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:55:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:55:41.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:55:41.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:55:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:55:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:55:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:55:41.704 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-17T23:55:41.701+0000 7f0114255640 -1 reset not still connected to 0x557d09ccdd40 2024-09-17T23:55:41.704 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-17T23:55:41.701+0000 7f0114255640 -1 reset not still connected to 0x557d09df6d00 2024-09-17T23:55:41.704 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-17T23:55:41.701+0000 7f0114255640 -1 reset not still connected to 0x557d0a065ee0 2024-09-17T23:55:41.704 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-17T23:55:41.702+0000 7f0114255640 -1 reset not still connected to 0x557d0b464750 2024-09-17T23:55:41.704 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-17T23:55:41.702+0000 7f0114255640 -1 reset not still connected to 0x557d0fabe0d0 2024-09-17T23:55:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:55:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:55:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:55:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:55:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:55:42.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:55:42.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1274 2024-09-17T23:55:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1275 2024-09-17T23:55:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1277 2024-09-17T23:55:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1276 2024-09-17T23:55:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:55:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:55:43.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:55:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:55:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:55:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:55:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:43.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:44.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1279 2024-09-17T23:55:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1278 2024-09-17T23:55:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:55:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:55:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:55:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:55:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:55:44.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:55:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1280 2024-09-17T23:55:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:55:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:55:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:55:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1281 2024-09-17T23:55:45.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:46.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1283 2024-09-17T23:55:46.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1282 2024-09-17T23:55:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:55:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:55:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:55:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:55:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:55:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:55:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:55:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:55:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:55:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:55:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:55:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:55:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:55:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:55:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:55:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1284 2024-09-17T23:55:47.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:48.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1285 2024-09-17T23:55:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1287 2024-09-17T23:55:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:48.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-17T23:55:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:55:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-17T23:55:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1286 2024-09-17T23:55:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:55:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:55:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:55:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1288 2024-09-17T23:55:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:55:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:55:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:55:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-17T23:55:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:55:49.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-17T23:55:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1291 2024-09-17T23:55:50.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1290 2024-09-17T23:55:50.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1289 2024-09-17T23:55:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:55:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:55:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:55:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1292 2024-09-17T23:55:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1293 2024-09-17T23:55:51.465 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:55:51.462+0000 7f9bfab49640 -1 reset not still connected to 0x55cd15d401a0 2024-09-17T23:55:51.465 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:55:51.462+0000 7f9bfab49640 -1 reset not still connected to 0x55cd20ad6dd0 2024-09-17T23:55:51.466 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:55:51.464+0000 7f9bfab49640 -1 reset not still connected to 0x55cd2a4576c0 2024-09-17T23:55:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:51.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:51.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-17T23:55:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:55:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-17T23:55:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:55:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:55:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:55:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-17T23:55:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:51.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:55:51.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-17T23:55:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1294 2024-09-17T23:55:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:55:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:55:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1295 2024-09-17T23:55:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1297 2024-09-17T23:55:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:55:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 71338 2024-09-17T23:55:53.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-17T23:55:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:55:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-17T23:55:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-17T23:55:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:55:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-17T23:55:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1298 2024-09-17T23:55:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1296 2024-09-17T23:55:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-17T23:55:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:55:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-17T23:55:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:55:54.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 71470 2024-09-17T23:55:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1299 2024-09-17T23:55:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1300 2024-09-17T23:55:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-17T23:55:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:55:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-17T23:55:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1301 2024-09-17T23:55:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-17T23:55:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:55:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-17T23:55:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1302 2024-09-17T23:55:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-17T23:55:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-17T23:55:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:55:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:55:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:55:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-17T23:55:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:55:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 71602 2024-09-17T23:55:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:55:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 71754 2024-09-17T23:55:57.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:55:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 71906 2024-09-17T23:55:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1303 2024-09-17T23:55:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:55:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 72059 2024-09-17T23:55:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:58.776 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:55:58.774+0000 7fc275b74640 -1 reset not still connected to 0x559cbfceb520 2024-09-17T23:55:58.776 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:55:58.774+0000 7fc275b74640 -1 reset not still connected to 0x559cc3ed0820 2024-09-17T23:55:58.777 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:55:58.774+0000 7fc275b74640 -1 reset not still connected to 0x559cc9206f70 2024-09-17T23:55:58.777 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:55:58.774+0000 7fc275b74640 -1 reset not still connected to 0x559ccd2cef70 2024-09-17T23:55:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1304 2024-09-17T23:55:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-17T23:55:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:55:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 72212 2024-09-17T23:56:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:56:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 72365 2024-09-17T23:56:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-17T23:56:01.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 72538 2024-09-17T23:56:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-17T23:56:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:56:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 71207 2024-09-17T23:57:29.092 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:57:29.089+0000 7f9bfab49640 -1 reset not still connected to 0x55cd20fd8a90 2024-09-17T23:58:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:58:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:58:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:58:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 71339 2024-09-17T23:58:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:58:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:58:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:58:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 71471 2024-09-17T23:58:26.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:58:26.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:58:26.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:58:26.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 71603 2024-09-17T23:58:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:58:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:58:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:58:27.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 71755 2024-09-17T23:58:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:58:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:58:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:58:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 71907 2024-09-17T23:58:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:58:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:58:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:58:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 72060 2024-09-17T23:58:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:58:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:58:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:58:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 72213 2024-09-17T23:58:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:58:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:58:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:58:32.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 72366 2024-09-17T23:58:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:58:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:58:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-17T23:58:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 72539 2024-09-17T23:58:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-17T23:58:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-17T23:58:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-17T23:58:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:58:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:58:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:58:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 1 2024-09-17T23:58:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:58:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd0 2024-09-17T23:58:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary1 2024-09-17T23:58:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:58:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary1 2024-09-17T23:58:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:58:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary1 2024-09-17T23:58:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 2 2024-09-17T23:58:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:58:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd1 2024-09-17T23:58:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:58:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 3 2024-09-17T23:58:34.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary2 2024-09-17T23:58:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary2 2024-09-17T23:58:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:58:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary2 2024-09-17T23:58:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd2 2024-09-17T23:58:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:58:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary3 2024-09-17T23:58:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:58:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary3 2024-09-17T23:58:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:58:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 4 2024-09-17T23:58:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary3 2024-09-17T23:58:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd3 2024-09-17T23:58:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary4 2024-09-17T23:58:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary4 2024-09-17T23:58:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:58:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:58:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary4 2024-09-17T23:58:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:58:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 5 2024-09-17T23:58:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd4 2024-09-17T23:58:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:58:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary5 2024-09-17T23:58:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary5 2024-09-17T23:58:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:58:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary5 2024-09-17T23:58:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:58:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 6 2024-09-17T23:58:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd5 2024-09-17T23:58:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary6 2024-09-17T23:58:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary6 2024-09-17T23:58:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:58:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:58:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary6 2024-09-17T23:58:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:58:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:58:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 7 2024-09-17T23:58:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:58:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:58:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd6 2024-09-17T23:58:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary7 2024-09-17T23:58:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary7 2024-09-17T23:58:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:58:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary7 2024-09-17T23:58:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:58:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-17T23:58:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:58:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 79365 2024-09-17T23:58:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 10 2024-09-17T23:58:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd9 2024-09-17T23:58:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary10 2024-09-17T23:58:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary10 2024-09-17T23:58:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:58:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary10 2024-09-17T23:58:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 8 2024-09-17T23:58:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 9 2024-09-17T23:58:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd8 2024-09-17T23:58:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd7 2024-09-17T23:58:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary9 2024-09-17T23:58:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary8 2024-09-17T23:58:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary9 2024-09-17T23:58:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary8 2024-09-17T23:58:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:58:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-17T23:58:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary9 2024-09-17T23:58:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary8 2024-09-17T23:58:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-17T23:58:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:58:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-17T23:58:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:58:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:58:34.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-17T23:58:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:58:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-17T23:58:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:58:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-17T23:58:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-17T23:58:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:58:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-17T23:58:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:58:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-17T23:58:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-17T23:58:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:58:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:58:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-17T23:58:34.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:59:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=310d719321d62df54330d415141acf86 2024-09-17T23:59:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-17T23:59:03.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-17T23:59:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:59:03.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-17T23:59:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=e62d5557f499c9cca6bc2d3c9b315e92 2024-09-17T23:59:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-17T23:59:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-17T23:59:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:59:03.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-17T23:59:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:59:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-17T23:59:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:59:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=3fd4316aa18b66006821f82cdf48e64c 2024-09-17T23:59:04.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-17T23:59:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-17T23:59:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:59:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-17T23:59:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=deafca769c14cc18a0614005ec4e2a65 2024-09-17T23:59:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-17T23:59:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=1f85a4eca4f3c9253fea464e20b65ec2 2024-09-17T23:59:05.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-17T23:59:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=c25f239457d054f6716f656e4b54d9c6 2024-09-17T23:59:05.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-17T23:59:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-17T23:59:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:59:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-17T23:59:05.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-17T23:59:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:59:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-17T23:59:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=8b10c5e625e5ce712444d77d244b2f6d 2024-09-17T23:59:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-17T23:59:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-17T23:59:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:59:05.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-17T23:59:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-17T23:59:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:59:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-17T23:59:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=03d02309b7b3b9c1cfb50b413cbe8319 2024-09-17T23:59:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-17T23:59:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=f7c7a679db2b78cd4c7872576138aef3 2024-09-17T23:59:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-17T23:59:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:59:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-17T23:59:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:59:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-17T23:59:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-17T23:59:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:59:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-17T23:59:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-17T23:59:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:59:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-17T23:59:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-17T23:59:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:59:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:59:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-17T23:59:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:59:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-17T23:59:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8171daf5-079e-42db-ab46-78768904f13b 2024-09-17T23:59:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:59:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:59:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:02 2024-09-17T23:59:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:05' 2024-09-17T23:59:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-17T23:59:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-17T23:59:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=7c805aeb2e10bc255f4e98a5ba87e434 2024-09-17T23:59:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-17T23:59:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-17T23:59:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:59:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-17T23:59:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:59:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:59:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:59:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-17T23:59:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:59:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:59:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-17T23:59:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:59:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-17T23:59:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:59:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:06.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:06.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-17T23:59:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-17T23:59:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:59:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:06.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-17T23:59:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:59:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-17T23:59:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:59:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-17T23:59:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-17T23:59:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1b62056c-37af-4377-8ef5-6fd1b5c24a62 2024-09-17T23:59:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:59:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:59:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:02 2024-09-17T23:59:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:06' 2024-09-17T23:59:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-17T23:59:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-17T23:59:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-17T23:59:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d50bf219-7b94-4aed-ac1f-03c36a7a6386 2024-09-17T23:59:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:59:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:59:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:02 2024-09-17T23:59:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:59:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2680832.0,"last_snapshot_bytes":5222400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617352,"remote_snapshot_timestamp":1726617352,"replay_state":"idle"} 2024-09-17T23:59:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:58:47' 2024-09-17T23:59:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-17T23:59:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-17T23:59:07.041 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:07.038+0000 7f9bfab49640 -1 reset not still connected to 0x55cd15d401a0 2024-09-17T23:59:07.041 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:07.038+0000 7f9bfab49640 -1 reset not still connected to 0x55cd20ad6dd0 2024-09-17T23:59:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:59:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:59:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-17T23:59:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-17T23:59:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:59:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-17T23:59:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:59:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:07.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-17T23:59:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:59:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-17T23:59:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:59:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-17T23:59:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-17T23:59:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:07.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:59:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-17T23:59:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-17T23:59:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:59:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-17T23:59:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-17T23:59:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:59:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-17T23:59:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-17T23:59:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9a4e684a-d528-43f1-9e0b-f06c8eaea047 2024-09-17T23:59:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:07.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-17T23:59:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:07.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:59:07.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:07.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:07.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:07.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-17T23:59:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 801c5de6-b3f3-4fca-ba00-b00b417252d2 2024-09-17T23:59:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:07.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-17T23:59:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:59:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:59:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:07.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:07.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-17T23:59:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 0b7bb811-ff64-4199-9949-17659e6a10ed 2024-09-17T23:59:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:59:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:59:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-17T23:59:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 7e11cb55-df54-4319-b4ee-da4847b7d252 2024-09-17T23:59:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:59:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:07.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:59:07.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:07.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:59:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:59:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-17T23:59:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8171daf5-079e-42db-ab46-78768904f13b 2024-09-17T23:59:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:59:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:59:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3424256.0,"last_snapshot_bytes":6709248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617349,"remote_snapshot_timestamp":1726617349,"replay_state":"idle"}2024-09-17 23:58:47image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4345856.0,"last_snapshot_bytes":8552448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617352,"remote_snapshot_timestamp":1726617352,"replay_state":"idle"}2024-09-17 23:58:47image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":927744.0,"last_snapshot_bytes":1667072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617355,"remote_snapshot_timestamp":1726617355,"replay_state":"idle"}2024-09-17 23:58:47image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07' 2024-09-17T23:59:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-17T23:59:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3424256.0,"last_snapshot_bytes":6709248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617349,"remote_snapshot_timestamp":1726617349,"replay_state":"idle"}2024-09-17 23:58:47image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4345856.0,"last_snapshot_bytes":8552448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617352,"remote_snapshot_timestamp":1726617352,"replay_state":"idle"}2024-09-17 23:58:47image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":927744.0,"last_snapshot_bytes":1667072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617355,"remote_snapshot_timestamp":1726617355,"replay_state":"idle"}2024-09-17 23:58:47image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07' 2024-09-17T23:59:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-17T23:59:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-17T23:59:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-17T23:59:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:08.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-17T23:59:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:59:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:08.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-17T23:59:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:59:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3424256.0,"last_snapshot_bytes":6709248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617349,"remote_snapshot_timestamp":1726617349,"replay_state":"idle"}2024-09-17 23:58:47image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4345856.0,"last_snapshot_bytes":8552448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617352,"remote_snapshot_timestamp":1726617352,"replay_state":"idle"}2024-09-17 23:58:47image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":927744.0,"last_snapshot_bytes":1667072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617355,"remote_snapshot_timestamp":1726617355,"replay_state":"idle"}2024-09-17 23:58:47image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07' 2024-09-17T23:59:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-17T23:59:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:08.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-17T23:59:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:08.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:08.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:08.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-17T23:59:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:59:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:08.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3424256.0,"last_snapshot_bytes":6709248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617349,"remote_snapshot_timestamp":1726617349,"replay_state":"idle"}2024-09-17 23:58:47image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4345856.0,"last_snapshot_bytes":8552448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617352,"remote_snapshot_timestamp":1726617352,"replay_state":"idle"}2024-09-17 23:58:47image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":927744.0,"last_snapshot_bytes":1667072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617355,"remote_snapshot_timestamp":1726617355,"replay_state":"idle"}2024-09-17 23:58:47image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07' 2024-09-17T23:59:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-17T23:59:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-17T23:59:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-17T23:59:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:59:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3424256.0,"last_snapshot_bytes":6709248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617349,"remote_snapshot_timestamp":1726617349,"replay_state":"idle"}2024-09-17 23:58:47image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4345856.0,"last_snapshot_bytes":8552448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617352,"remote_snapshot_timestamp":1726617352,"replay_state":"idle"}2024-09-17 23:58:47image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":927744.0,"last_snapshot_bytes":1667072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617355,"remote_snapshot_timestamp":1726617355,"replay_state":"idle"}2024-09-17 23:58:47image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07' 2024-09-17T23:59:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-17T23:59:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-17T23:59:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:08.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:08.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-17T23:59:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:59:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:08.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:08.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-17T23:59:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:59:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-17T23:59:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:59:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-17T23:59:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:08.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-17T23:59:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-17T23:59:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:59:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-17T23:59:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-17T23:59:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 24918fa8-5a55-499c-98f9-3d026778b128 2024-09-17T23:59:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:08' 2024-09-17T23:59:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-17T23:59:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 04e5f6d7-d750-4252-b6b3-7e6d97d62e06 2024-09-17T23:59:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:08' 2024-09-17T23:59:08.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:59:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:08.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:08.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:59:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-17T23:59:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 07a8d24c-e200-4a35-a6cb-a1ab753f27bf 2024-09-17T23:59:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:08' 2024-09-17T23:59:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:59:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:08.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:59:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:59:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:59:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-17T23:59:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:59:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-17T23:59:08.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:08image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:08image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:08image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07' 2024-09-17T23:59:08.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-17T23:59:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-17T23:59:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:08image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:08image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:08image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07' 2024-09-17T23:59:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-17T23:59:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-17T23:59:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:59:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-17T23:59:08.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:08.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:08.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:08image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:08image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:08image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07' 2024-09-17T23:59:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-17T23:59:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-17T23:59:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:59:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-17T23:59:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:08.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-17T23:59:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:59:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:08.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-17T23:59:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1b62056c-37af-4377-8ef5-6fd1b5c24a62 2024-09-17T23:59:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:08.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:59:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:08.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:08.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:59:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:08.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:08.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-17T23:59:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:59:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:08.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-17T23:59:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:08image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:08image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:08image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07' 2024-09-17T23:59:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:09.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-17T23:59:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:09.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-17T23:59:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-17T23:59:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d50bf219-7b94-4aed-ac1f-03c36a7a6386 2024-09-17T23:59:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-17T23:59:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:59:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:59:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:09.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:59:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:09.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:08image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:08image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:08image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image is not primary42634266mirror.0smithi1922024-09-17 23:59:07cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-17 23:59:07' 2024-09-17T23:59:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-17T23:59:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-17T23:59:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-17T23:59:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:59:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-17T23:59:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:59:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-17T23:59:09.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-17T23:59:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:59:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-17T23:59:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-17T23:59:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9a4e684a-d528-43f1-9e0b-f06c8eaea047 2024-09-17T23:59:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:59:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:09.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:09.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:59:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-17T23:59:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 801c5de6-b3f3-4fca-ba00-b00b417252d2 2024-09-17T23:59:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:59:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:59:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:59:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-17T23:59:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:59:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-17T23:59:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:59:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:09.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:09.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-17T23:59:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:59:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-17T23:59:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-17T23:59:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 7e11cb55-df54-4319-b4ee-da4847b7d252 2024-09-17T23:59:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:59:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:59:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:09.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:09.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:09.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:09.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:59:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-17T23:59:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:59:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-17T23:59:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07' 2024-09-17T23:59:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-17T23:59:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-17T23:59:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 0b7bb811-ff64-4199-9949-17659e6a10ed 2024-09-17T23:59:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:59:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07' 2024-09-17T23:59:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-17T23:59:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-17T23:59:09.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:59:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-17T23:59:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-17T23:59:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-17T23:59:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:59:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-17T23:59:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-17T23:59:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-17T23:59:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:59:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-17T23:59:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:59:09.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:09.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-17T23:59:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8171daf5-079e-42db-ab46-78768904f13b 2024-09-17T23:59:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:59:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07' 2024-09-17T23:59:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:09.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-17T23:59:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:59:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-17T23:59:09.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-17T23:59:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-17T23:59:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:59:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-17T23:59:09.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07' 2024-09-17T23:59:09.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:09.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-17T23:59:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:09.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-17T23:59:09.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:09.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:09.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:09.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-17T23:59:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:59:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:59:09.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-17T23:59:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07' 2024-09-17T23:59:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-17T23:59:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-17T23:59:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-17T23:59:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:59:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:59:09.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-17T23:59:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:59:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:59:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:59:09.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:59:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:59:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:59:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-17T23:59:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:59:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:59:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:59:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:59:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-17T23:59:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-17T23:59:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:59:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:59:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-17T23:59:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd1 2024-09-17T23:59:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd2 2024-09-17T23:59:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd0 2024-09-17T23:59:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-17T23:59:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-17T23:59:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:59:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:59:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-17T23:59:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:59:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd3 2024-09-17T23:59:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-17T23:59:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:59:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-17T23:59:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:59:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-17T23:59:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-17T23:59:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:59:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-17T23:59:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-17T23:59:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:59:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-17T23:59:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-17T23:59:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 24918fa8-5a55-499c-98f9-3d026778b128 2024-09-17T23:59:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:10.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:08 2024-09-17T23:59:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:59:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:10.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:10.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:59:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:10.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-17T23:59:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 07a8d24c-e200-4a35-a6cb-a1ab753f27bf 2024-09-17T23:59:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:08 2024-09-17T23:59:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:59:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:08' 2024-09-17T23:59:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-17 23:59:08' 2024-09-17T23:59:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:59:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:08' 2024-09-17T23:59:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-17T23:59:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 04e5f6d7-d750-4252-b6b3-7e6d97d62e06 2024-09-17T23:59:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:08 2024-09-17T23:59:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:59:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:59:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:59:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:08' 2024-09-17T23:59:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-17 23:59:08' 2024-09-17T23:59:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:59:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:08' 2024-09-17T23:59:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:08' 2024-09-17T23:59:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-17 23:59:08' 2024-09-17T23:59:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:59:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:08' 2024-09-17T23:59:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-17T23:59:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:59:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-17T23:59:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-17T23:59:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:59:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-17T23:59:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1b62056c-37af-4377-8ef5-6fd1b5c24a62 2024-09-17T23:59:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:59:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-17T23:59:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-17T23:59:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:10.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:10.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:10.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:10.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:10.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:59:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07' 2024-09-17T23:59:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-17T23:59:10.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:08' 2024-09-17T23:59:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07' 2024-09-17T23:59:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-17T23:59:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-17T23:59:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:08' 2024-09-17T23:59:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-17T23:59:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:10.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:10.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:10.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-17T23:59:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:59:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:59:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-17T23:59:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07' 2024-09-17T23:59:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-17T23:59:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-17T23:59:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:59:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:59:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-17T23:59:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:08' 2024-09-17T23:59:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-17T23:59:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:10.283 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:59:10.281+0000 7fc275b74640 -1 reset not still connected to 0x559cc9346b60 2024-09-17T23:59:10.283 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-17T23:59:10.281+0000 7fc275b74640 -1 reset not still connected to 0x559cc9347040 2024-09-17T23:59:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-17T23:59:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:59:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:59:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-17T23:59:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-17T23:59:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d50bf219-7b94-4aed-ac1f-03c36a7a6386 2024-09-17T23:59:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-17T23:59:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07 2024-09-17T23:59:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-17T23:59:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-17T23:59:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-17T23:59:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:07' 2024-09-17T23:59:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-17T23:59:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-17T23:59:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-17T23:59:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:10.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:10.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-17 23:59:07' 2024-09-17T23:59:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:59:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-17T23:59:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:07' 2024-09-17T23:59:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:10.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07' 2024-09-17T23:59:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-17T23:59:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-17T23:59:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:10.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-17T23:59:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:59:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:59:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-17T23:59:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary58171daf5-079e-42db-ab46-78768904f13bup+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+unknownremote image is not primary42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:08cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+unknownremote image demoted42604266mirror.0smithi1922024-09-17 23:59:07cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-17 23:59:07' 2024-09-17T23:59:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-17T23:59:10.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:07' 2024-09-17T23:59:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-17T23:59:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-17T23:59:10.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-17T23:59:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-17T23:59:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-17T23:59:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-17T23:59:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:59:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-17T23:59:10.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-17T23:59:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:59:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:59:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:59:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:59:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:59:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-17T23:59:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:59:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:59:10.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-17T23:59:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:59:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:59:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-17T23:59:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:59:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:59:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:59:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:59:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-17T23:59:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-17T23:59:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:59:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:59:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-17T23:59:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-17T23:59:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-17T23:59:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-17T23:59:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd4 2024-09-17T23:59:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-17T23:59:10.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:59:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd6 2024-09-17T23:59:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:59:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-17T23:59:10.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd5 2024-09-17T23:59:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:59:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-17T23:59:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd7 2024-09-17T23:59:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:59:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-17T23:59:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd9 2024-09-17T23:59:11.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:59:11.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd8 2024-09-17T23:59:11.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-17T23:59:11.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-17T23:59:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-17T23:59:12.486 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:59:12.484+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4dd0 2024-09-17T23:59:12.487 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:59:12.485+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb5790 2024-09-17T23:59:12.487 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:59:12.485+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ea270 2024-09-17T23:59:12.488 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:59:12.487+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-17T23:59:12.489 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-17T23:59:12.487+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-17T23:59:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=c25f239457d054f6716f656e4b54d9c6 2024-09-17T23:59:39.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-17T23:59:39.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ c25f239457d054f6716f656e4b54d9c6 != \c\2\5\f\2\3\9\4\5\7\d\0\5\4\f\6\7\1\6\f\6\5\6\e\4\b\5\4\d\9\c\6 ]] 2024-09-17T23:59:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=deafca769c14cc18a0614005ec4e2a65 2024-09-17T23:59:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-17T23:59:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ deafca769c14cc18a0614005ec4e2a65 != \d\e\a\f\c\a\7\6\9\c\1\4\c\c\1\8\a\0\6\1\4\0\0\5\e\c\4\e\2\a\6\5 ]] 2024-09-17T23:59:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=3fd4316aa18b66006821f82cdf48e64c 2024-09-17T23:59:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-17T23:59:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 3fd4316aa18b66006821f82cdf48e64c != \3\f\d\4\3\1\6\a\a\1\8\b\6\6\0\0\6\8\2\1\f\8\2\c\d\f\4\8\e\6\4\c ]] 2024-09-17T23:59:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=8b10c5e625e5ce712444d77d244b2f6d 2024-09-17T23:59:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-17T23:59:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 8b10c5e625e5ce712444d77d244b2f6d != \8\b\1\0\c\5\e\6\2\5\e\5\c\e\7\1\2\4\4\4\d\7\7\d\2\4\4\b\2\f\6\d ]] 2024-09-17T23:59:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=03d02309b7b3b9c1cfb50b413cbe8319 2024-09-17T23:59:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-17T23:59:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=1f85a4eca4f3c9253fea464e20b65ec2 2024-09-17T23:59:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-17T23:59:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 03d02309b7b3b9c1cfb50b413cbe8319 != \0\3\d\0\2\3\0\9\b\7\b\3\b\9\c\1\c\f\b\5\0\b\4\1\3\c\b\e\8\3\1\9 ]] 2024-09-17T23:59:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:59:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 79366 2024-09-17T23:59:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:59:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 79367 2024-09-17T23:59:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=310d719321d62df54330d415141acf86 2024-09-17T23:59:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-17T23:59:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 1f85a4eca4f3c9253fea464e20b65ec2 != \1\f\8\5\a\4\e\c\a\4\f\3\c\9\2\5\3\f\e\a\4\6\4\e\2\0\b\6\5\e\c\2 ]] 2024-09-17T23:59:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:59:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 79369 2024-09-17T23:59:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 310d719321d62df54330d415141acf86 != \3\1\0\d\7\1\9\3\2\1\d\6\2\d\f\5\4\3\3\0\d\4\1\5\1\4\1\a\c\f\8\6 ]] 2024-09-17T23:59:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=7c805aeb2e10bc255f4e98a5ba87e434 2024-09-17T23:59:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-17T23:59:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 7c805aeb2e10bc255f4e98a5ba87e434 != \7\c\8\0\5\a\e\b\2\e\1\0\b\c\2\5\5\f\4\e\9\8\a\5\b\a\8\7\e\4\3\4 ]] 2024-09-17T23:59:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=f7c7a679db2b78cd4c7872576138aef3 2024-09-17T23:59:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-17T23:59:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ f7c7a679db2b78cd4c7872576138aef3 != \f\7\c\7\a\6\7\9\d\b\2\b\7\8\c\d\4\c\7\8\7\2\5\7\6\1\3\8\a\e\f\3 ]] 2024-09-17T23:59:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:59:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 79372 2024-09-17T23:59:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:59:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 79374 2024-09-17T23:59:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=e62d5557f499c9cca6bc2d3c9b315e92 2024-09-17T23:59:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-17T23:59:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ e62d5557f499c9cca6bc2d3c9b315e92 != \e\6\2\d\5\5\5\7\f\4\9\9\c\9\c\c\a\6\b\c\2\d\3\c\9\b\3\1\5\e\9\2 ]] 2024-09-17T23:59:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:59:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 79376 2024-09-17T23:59:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:59:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 79378 2024-09-17T23:59:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:59:43.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 79379 2024-09-17T23:59:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-17T23:59:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 79381 2024-09-17T23:59:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-17T23:59:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:59:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-17T23:59:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-17T23:59:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:59:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:59:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-17T23:59:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:43.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:59:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-17T23:59:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:59:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-17T23:59:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-17T23:59:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 24918fa8-5a55-499c-98f9-3d026778b128 2024-09-17T23:59:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:59:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":15011840.0,"bytes_per_snapshot":61214720.0,"last_snapshot_bytes":61214720,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"} 2024-09-17T23:59:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:17 2024-09-17T23:59:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:59:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:59:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:42' 2024-09-17T23:59:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:59:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:59:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":15011840.0,"bytes_per_snapshot":61214720.0,"last_snapshot_bytes":61214720,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"} 2024-09-17T23:59:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:59:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:59:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:17' 2024-09-17T23:59:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-17 23:59:17' 2024-09-17T23:59:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:59:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:17' 2024-09-17T23:59:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:44.458 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:44.456+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1d737e10 2024-09-17T23:59:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary124918fa8-5a55-499c-98f9-3d026778b128up+replayingreplaying, {"bytes_per_second":15011840.0,"bytes_per_snapshot":61214720.0,"last_snapshot_bytes":61214720,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+replayingreplaying, {"bytes_per_second":35962880.0,"bytes_per_snapshot":65273856.0,"last_snapshot_bytes":65273856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+replayingreplaying, {"bytes_per_second":27506688.0,"bytes_per_snapshot":62881792.0,"last_snapshot_bytes":62881792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+replayingreplaying, {"bytes_per_second":22499328.0,"bytes_per_snapshot":61091840.0,"last_snapshot_bytes":61091840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+replayingreplaying, {"bytes_per_second":25380864.0,"bytes_per_snapshot":61169664.0,"last_snapshot_bytes":61169664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary58171daf5-079e-42db-ab46-78768904f13bup+replayingreplaying, {"bytes_per_second":18259968.0,"bytes_per_snapshot":60235776.0,"last_snapshot_bytes":60235776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+replayingreplaying, {"bytes_per_second":57020416.0,"bytes_per_snapshot":71516160.0,"last_snapshot_bytes":71516160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+replayingreplaying, {"bytes_per_second":35835904.0,"bytes_per_snapshot":71671808.0,"last_snapshot_bytes":71671808,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+replayingreplaying, {"bytes_per_second":10250240.0,"bytes_per_snapshot":62930944.0,"last_snapshot_bytes":62930944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+replayingreplaying, {"bytes_per_second":17678336.0,"bytes_per_snapshot":62984192.0,"last_snapshot_bytes":62984192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42' 2024-09-17T23:59:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-17T23:59:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:17' 2024-09-17T23:59:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-17T23:59:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:59:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:59:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:59:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:59:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-17T23:59:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:59:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-17T23:59:44.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-17T23:59:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:59:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-17T23:59:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-17T23:59:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:45.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:59:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary1 2024-09-17T23:59:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary1 2024-09-17T23:59:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:59:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-17T23:59:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-17T23:59:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:59:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:59:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-17T23:59:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:45.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:59:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-17T23:59:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:59:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-17T23:59:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-17T23:59:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 0b7bb811-ff64-4199-9949-17659e6a10ed 2024-09-17T23:59:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:59:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":27506688.0,"bytes_per_snapshot":62881792.0,"last_snapshot_bytes":62881792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"} 2024-09-17T23:59:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:17 2024-09-17T23:59:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:59:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:59:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:42' 2024-09-17T23:59:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:59:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:59:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":27506688.0,"bytes_per_snapshot":62881792.0,"last_snapshot_bytes":62881792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"} 2024-09-17T23:59:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:59:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:59:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:17' 2024-09-17T23:59:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-17 23:59:17' 2024-09-17T23:59:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:17' 2024-09-17T23:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK942634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+replayingreplaying, {"bytes_per_second":35962880.0,"bytes_per_snapshot":65273856.0,"last_snapshot_bytes":65273856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary20b7bb811-ff64-4199-9949-17659e6a10edup+replayingreplaying, {"bytes_per_second":27506688.0,"bytes_per_snapshot":62881792.0,"last_snapshot_bytes":62881792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+replayingreplaying, {"bytes_per_second":22499328.0,"bytes_per_snapshot":61091840.0,"last_snapshot_bytes":61091840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+replayingreplaying, {"bytes_per_second":25380864.0,"bytes_per_snapshot":61169664.0,"last_snapshot_bytes":61169664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary58171daf5-079e-42db-ab46-78768904f13bup+replayingreplaying, {"bytes_per_second":18259968.0,"bytes_per_snapshot":60235776.0,"last_snapshot_bytes":60235776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+replayingreplaying, {"bytes_per_second":57020416.0,"bytes_per_snapshot":71516160.0,"last_snapshot_bytes":71516160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+replayingreplaying, {"bytes_per_second":35835904.0,"bytes_per_snapshot":71671808.0,"last_snapshot_bytes":71671808,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+replayingreplaying, {"bytes_per_second":10250240.0,"bytes_per_snapshot":62930944.0,"last_snapshot_bytes":62930944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+replayingreplaying, {"bytes_per_second":17678336.0,"bytes_per_snapshot":62984192.0,"last_snapshot_bytes":62984192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:42' 2024-09-17T23:59:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-17T23:59:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:17' 2024-09-17T23:59:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-17T23:59:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:59:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:59:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:59:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:59:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-17T23:59:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:59:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-17T23:59:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-17T23:59:47.939 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:47.936+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0b849d40 2024-09-17T23:59:47.940 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:47.937+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0c3d0340 2024-09-17T23:59:47.943 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:47.938+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1feb7860 2024-09-17T23:59:47.943 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:47.940+0000 7f9bfab49640 -1 reset not still connected to 0x55cd2617c000 2024-09-17T23:59:48.011 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:48.008+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-17T23:59:48.012 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:48.009+0000 7f9bfab49640 -1 reset not still connected to 0x55cd08f7ed00 2024-09-17T23:59:48.012 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:48.009+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0b849d40 2024-09-17T23:59:48.014 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:48.009+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0c3d0340 2024-09-17T23:59:48.014 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:48.012+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1feb7860 2024-09-17T23:59:48.015 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:48.013+0000 7f9bfab49640 -1 reset not still connected to 0x55cd2617c000 2024-09-17T23:59:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:59:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-17T23:59:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-17T23:59:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:59:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary2 2024-09-17T23:59:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary2 2024-09-17T23:59:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:59:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-17T23:59:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-17T23:59:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:59:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:59:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-17T23:59:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:59:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:49.411 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:49.406+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782eea0 2024-09-17T23:59:49.411 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:49.406+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-17T23:59:49.412 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:49.408+0000 7f9bfab49640 -1 reset not still connected to 0x55cd08f7ed00 2024-09-17T23:59:49.412 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:49.408+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0b849d40 2024-09-17T23:59:49.412 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:49.408+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0c3d0340 2024-09-17T23:59:49.412 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-17T23:59:49.408+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1feb7860 2024-09-17T23:59:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-17T23:59:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:59:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-17T23:59:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-17T23:59:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d50bf219-7b94-4aed-ac1f-03c36a7a6386 2024-09-17T23:59:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:59:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2036394.67,"bytes_per_snapshot":61091840.0,"last_snapshot_bytes":61091840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"} 2024-09-17T23:59:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:47 2024-09-17T23:59:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:59:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:59:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:47' 2024-09-17T23:59:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:59:49.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:59:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:49.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2036394.67,"bytes_per_snapshot":61091840.0,"last_snapshot_bytes":61091840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"} 2024-09-17T23:59:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:59:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:59:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-17T23:59:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-17 23:59:47' 2024-09-17T23:59:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:59:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:47' 2024-09-17T23:59:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:50.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK842634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+replayingreplaying, {"bytes_per_second":2175795.2,"bytes_per_snapshot":65273856.0,"last_snapshot_bytes":65273856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:47image-primary3d50bf219-7b94-4aed-ac1f-03c36a7a6386up+replayingreplaying, {"bytes_per_second":2036394.67,"bytes_per_snapshot":61091840.0,"last_snapshot_bytes":61091840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:47image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+replayingreplaying, {"bytes_per_second":2038988.8,"bytes_per_snapshot":61169664.0,"last_snapshot_bytes":61169664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:47image-primary58171daf5-079e-42db-ab46-78768904f13bup+replayingreplaying, {"bytes_per_second":2007859.2,"bytes_per_snapshot":60235776.0,"last_snapshot_bytes":60235776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:47image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+replayingreplaying, {"bytes_per_second":2383872.0,"bytes_per_snapshot":71516160.0,"last_snapshot_bytes":71516160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:47image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+replayingreplaying, {"bytes_per_second":2389060.27,"bytes_per_snapshot":71671808.0,"last_snapshot_bytes":71671808,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:47image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+replayingreplaying, {"bytes_per_second":2097698.13,"bytes_per_snapshot":62930944.0,"last_snapshot_bytes":62930944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:47image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+replayingreplaying, {"bytes_per_second":2099473.07,"bytes_per_snapshot":62984192.0,"last_snapshot_bytes":62984192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:47' 2024-09-17T23:59:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:50.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-17T23:59:50.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-17T23:59:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-17T23:59:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:59:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:59:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:59:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:59:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-17T23:59:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:59:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-17T23:59:50.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-17T23:59:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:59:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-17T23:59:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-17T23:59:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:59:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary3 2024-09-17T23:59:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary3 2024-09-17T23:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-17T23:59:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-17T23:59:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:59:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:59:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-17T23:59:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:59:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-17T23:59:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:59:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-17T23:59:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-17T23:59:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 07a8d24c-e200-4a35-a6cb-a1ab753f27bf 2024-09-17T23:59:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:59:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2038988.8,"bytes_per_snapshot":61169664.0,"last_snapshot_bytes":61169664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"} 2024-09-17T23:59:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:47 2024-09-17T23:59:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:59:53.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:59:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:52' 2024-09-17T23:59:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:59:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:59:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2038988.8,"bytes_per_snapshot":61169664.0,"last_snapshot_bytes":61169664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"} 2024-09-17T23:59:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:59:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:59:53.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-17T23:59:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-17 23:59:47' 2024-09-17T23:59:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:59:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:47' 2024-09-17T23:59:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK742634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+replayingreplaying, {"bytes_per_second":2175795.2,"bytes_per_snapshot":65273856.0,"last_snapshot_bytes":65273856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:52image-primary407a8d24c-e200-4a35-a6cb-a1ab753f27bfup+replayingreplaying, {"bytes_per_second":2038988.8,"bytes_per_snapshot":61169664.0,"last_snapshot_bytes":61169664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:52image-primary58171daf5-079e-42db-ab46-78768904f13bup+replayingreplaying, {"bytes_per_second":2007859.2,"bytes_per_snapshot":60235776.0,"last_snapshot_bytes":60235776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:52image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+replayingreplaying, {"bytes_per_second":2383872.0,"bytes_per_snapshot":71516160.0,"last_snapshot_bytes":71516160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:52image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+replayingreplaying, {"bytes_per_second":2389060.27,"bytes_per_snapshot":71671808.0,"last_snapshot_bytes":71671808,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:52image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+replayingreplaying, {"bytes_per_second":2097698.13,"bytes_per_snapshot":62930944.0,"last_snapshot_bytes":62930944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:52image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+replayingreplaying, {"bytes_per_second":2099473.07,"bytes_per_snapshot":62984192.0,"last_snapshot_bytes":62984192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:52' 2024-09-17T23:59:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:53.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-17T23:59:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-17T23:59:53.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-17T23:59:53.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:59:53.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:59:53.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:59:53.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:59:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-17T23:59:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:59:53.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-17T23:59:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-17T23:59:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:59:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-17T23:59:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-17T23:59:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:59:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary4 2024-09-17T23:59:54.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:54.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary4 2024-09-17T23:59:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:59:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-17T23:59:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-17T23:59:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:59:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:59:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-17T23:59:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:59:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-17T23:59:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:59:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-17T23:59:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-17T23:59:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8171daf5-079e-42db-ab46-78768904f13b 2024-09-17T23:59:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:59:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2007859.2,"bytes_per_snapshot":60235776.0,"last_snapshot_bytes":60235776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"} 2024-09-17T23:59:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:47 2024-09-17T23:59:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:59:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:59:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:52' 2024-09-17T23:59:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:59:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:59:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2007859.2,"bytes_per_snapshot":60235776.0,"last_snapshot_bytes":60235776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"} 2024-09-17T23:59:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:59:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:59:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-17T23:59:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-17 23:59:47' 2024-09-17T23:59:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:59:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:47' 2024-09-17T23:59:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK642634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+replayingreplaying, {"bytes_per_second":2175795.2,"bytes_per_snapshot":65273856.0,"last_snapshot_bytes":65273856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:52image-primary58171daf5-079e-42db-ab46-78768904f13bup+replayingreplaying, {"bytes_per_second":2007859.2,"bytes_per_snapshot":60235776.0,"last_snapshot_bytes":60235776,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:52image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+replayingreplaying, {"bytes_per_second":2383872.0,"bytes_per_snapshot":71516160.0,"last_snapshot_bytes":71516160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:52image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+replayingreplaying, {"bytes_per_second":2389060.27,"bytes_per_snapshot":71671808.0,"last_snapshot_bytes":71671808,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:52image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+replayingreplaying, {"bytes_per_second":2097698.13,"bytes_per_snapshot":62930944.0,"last_snapshot_bytes":62930944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:52image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+replayingreplaying, {"bytes_per_second":2099473.07,"bytes_per_snapshot":62984192.0,"last_snapshot_bytes":62984192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:52' 2024-09-17T23:59:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-17T23:59:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-17T23:59:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-17T23:59:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:59:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:59:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:59:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:59:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-17T23:59:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:59:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-17T23:59:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-17T23:59:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:59:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-17T23:59:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-17T23:59:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:59:57.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary5 2024-09-17T23:59:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary5 2024-09-17T23:59:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:59:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-17T23:59:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-17T23:59:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:59:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:59:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-17T23:59:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:59:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-17T23:59:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-17T23:59:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:59:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-17T23:59:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-17T23:59:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1b62056c-37af-4377-8ef5-6fd1b5c24a62 2024-09-17T23:59:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-17T23:59:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2383872.0,"bytes_per_snapshot":71516160.0,"last_snapshot_bytes":71516160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"} 2024-09-17T23:59:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-17T23:59:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:47 2024-09-17T23:59:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-17T23:59:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-17T23:59:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-17T23:59:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-17T23:59:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:57' 2024-09-17T23:59:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-17T23:59:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-17T23:59:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-17T23:59:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2383872.0,"bytes_per_snapshot":71516160.0,"last_snapshot_bytes":71516160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"} 2024-09-17T23:59:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-17T23:59:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:59:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-17T23:59:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:59:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-17T23:59:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-17T23:59:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-17T23:59:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-17T23:59:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-17T23:59:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-17 23:59:47' 2024-09-17T23:59:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:59:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:47' 2024-09-17T23:59:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-17T23:59:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-17T23:59:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-17T23:59:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK542634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+replayingreplaying, {"bytes_per_second":2175795.2,"bytes_per_snapshot":65273856.0,"last_snapshot_bytes":65273856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:57image-primary61b62056c-37af-4377-8ef5-6fd1b5c24a62up+replayingreplaying, {"bytes_per_second":2383872.0,"bytes_per_snapshot":71516160.0,"last_snapshot_bytes":71516160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:57image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+replayingreplaying, {"bytes_per_second":2389060.27,"bytes_per_snapshot":71671808.0,"last_snapshot_bytes":71671808,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:57image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+replayingreplaying, {"bytes_per_second":2097698.13,"bytes_per_snapshot":62930944.0,"last_snapshot_bytes":62930944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:57image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+replayingreplaying, {"bytes_per_second":2099473.07,"bytes_per_snapshot":62984192.0,"last_snapshot_bytes":62984192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:57' 2024-09-17T23:59:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-17T23:59:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-17T23:59:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-17T23:59:58.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-17T23:59:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-17T23:59:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-17T23:59:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-17T23:59:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-17T23:59:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-17T23:59:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-17T23:59:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:59:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-17T23:59:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-17T23:59:59.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-17T23:59:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-17T23:59:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-17T23:59:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-17T23:59:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary6 2024-09-17T23:59:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-17T23:59:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary6 2024-09-17T23:59:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-17T23:59:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-17T23:59:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-17T23:59:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:59:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-17T23:59:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-17T23:59:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-17T23:59:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-17T23:59:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-17T23:59:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-17T23:59:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-17T23:59:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-17T23:59:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-17T23:59:59.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:00:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-18T00:00:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:00:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:00:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:00:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:00:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:00:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:00:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:00:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 04e5f6d7-d750-4252-b6b3-7e6d97d62e06 2024-09-18T00:00:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:00:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2389060.27,"bytes_per_snapshot":71671808.0,"last_snapshot_bytes":71671808,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"} 2024-09-18T00:00:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:00:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:47 2024-09-18T00:00:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:00:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:00:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:00:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:00:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:57' 2024-09-18T00:00:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:00:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:00:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:00:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2389060.27,"bytes_per_snapshot":71671808.0,"last_snapshot_bytes":71671808,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"} 2024-09-18T00:00:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:00:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:00:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:00:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:00:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:00:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:00:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:00:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:00:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-18T00:00:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-17 23:59:47' 2024-09-18T00:00:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:00:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:47' 2024-09-18T00:00:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:00:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:00:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:00:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+replayingreplaying, {"bytes_per_second":2175795.2,"bytes_per_snapshot":65273856.0,"last_snapshot_bytes":65273856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:57image-primary704e5f6d7-d750-4252-b6b3-7e6d97d62e06up+replayingreplaying, {"bytes_per_second":2389060.27,"bytes_per_snapshot":71671808.0,"last_snapshot_bytes":71671808,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617550,"remote_snapshot_timestamp":1726617550,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:57image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+replayingreplaying, {"bytes_per_second":2097698.13,"bytes_per_snapshot":62930944.0,"last_snapshot_bytes":62930944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:57image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+replayingreplaying, {"bytes_per_second":2099473.07,"bytes_per_snapshot":62984192.0,"last_snapshot_bytes":62984192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-17 23:59:57' 2024-09-18T00:00:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:00:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:00:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-18T00:00:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:00:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:00:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:00:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:00:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:00:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-18T00:00:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:00:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-18T00:00:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-18T00:00:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:00:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-18T00:00:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:00:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:00:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary7 2024-09-18T00:00:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary7 2024-09-18T00:00:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:00:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-18T00:00:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-18T00:00:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:00:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-18T00:00:02.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:00:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:00:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:00:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:00:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:00:03.804 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:00:03.802+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-18T00:00:03.805 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:00:03.802+0000 7f9bfab49640 -1 reset not still connected to 0x55cd08f7ed00 2024-09-18T00:00:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-18T00:00:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:00:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:00:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:00:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:00:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:00:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:00:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:00:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9a4e684a-d528-43f1-9e0b-f06c8eaea047 2024-09-18T00:00:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:00:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2097698.13,"bytes_per_snapshot":62930944.0,"last_snapshot_bytes":62930944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"} 2024-09-18T00:00:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:00:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:47 2024-09-18T00:00:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:00:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:00:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:00:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:00:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:00:02' 2024-09-18T00:00:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:00:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:00:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:00:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2097698.13,"bytes_per_snapshot":62930944.0,"last_snapshot_bytes":62930944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"} 2024-09-18T00:00:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:00:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:00:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:00:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:00:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:00:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:00:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:00:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:00:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-18T00:00:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-17 23:59:47' 2024-09-18T00:00:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:00:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:47' 2024-09-18T00:00:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:00:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:00:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:00:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK342634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+replayingreplaying, {"bytes_per_second":2175795.2,"bytes_per_snapshot":65273856.0,"last_snapshot_bytes":65273856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:00:02image-primary89a4e684a-d528-43f1-9e0b-f06c8eaea047up+replayingreplaying, {"bytes_per_second":2097698.13,"bytes_per_snapshot":62930944.0,"last_snapshot_bytes":62930944,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:00:02image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+replayingreplaying, {"bytes_per_second":2099473.07,"bytes_per_snapshot":62984192.0,"last_snapshot_bytes":62984192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:00:02' 2024-09-18T00:00:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:00:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:00:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-18T00:00:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:00:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:00:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:00:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:00:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:00:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-18T00:00:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:00:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-18T00:00:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-18T00:00:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:00:05.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-18T00:00:05.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:05.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:00:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:00:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary8 2024-09-18T00:00:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary8 2024-09-18T00:00:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:00:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-18T00:00:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-18T00:00:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:00:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-18T00:00:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:00:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:00:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:00:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:00:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:00:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-18T00:00:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:00:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:00:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:00:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:00:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:00:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:00:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:00:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 7e11cb55-df54-4319-b4ee-da4847b7d252 2024-09-18T00:00:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:00:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2099473.07,"bytes_per_snapshot":62984192.0,"last_snapshot_bytes":62984192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"} 2024-09-18T00:00:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:00:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:47 2024-09-18T00:00:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:00:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:00:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:00:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:00:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:00:02' 2024-09-18T00:00:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:00:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:00:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:00:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2099473.07,"bytes_per_snapshot":62984192.0,"last_snapshot_bytes":62984192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"} 2024-09-18T00:00:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:00:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:00:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:00:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:00:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:00:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:00:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:00:06.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:00:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-18T00:00:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-17 23:59:47' 2024-09-18T00:00:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:00:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:47' 2024-09-18T00:00:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:00:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:00:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:00:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK242634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+replayingreplaying, {"bytes_per_second":2175795.2,"bytes_per_snapshot":65273856.0,"last_snapshot_bytes":65273856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:00:02image-primary97e11cb55-df54-4319-b4ee-da4847b7d252up+replayingreplaying, {"bytes_per_second":2099473.07,"bytes_per_snapshot":62984192.0,"last_snapshot_bytes":62984192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:00:02' 2024-09-18T00:00:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:00:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:00:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-18T00:00:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:00:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:00:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:00:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:00:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:00:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-18T00:00:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:00:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-18T00:00:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-18T00:00:07.969 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:00:07.967+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782eea0 2024-09-18T00:00:07.969 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:00:07.967+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-18T00:00:07.972 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:00:07.970+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1e7a2820 2024-09-18T00:00:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:00:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-18T00:00:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:08.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:08.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:00:08.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:00:08.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary9 2024-09-18T00:00:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:08.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary9 2024-09-18T00:00:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:00:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-18T00:00:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-18T00:00:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:00:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-18T00:00:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:00:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:00:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:00:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:00:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:00:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-18T00:00:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:00:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:00:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:00:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:00:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:00:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:00:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:00:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 801c5de6-b3f3-4fca-ba00-b00b417252d2 2024-09-18T00:00:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:00:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2175795.2,"bytes_per_snapshot":65273856.0,"last_snapshot_bytes":65273856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"} 2024-09-18T00:00:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:00:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-17 23:59:47 2024-09-18T00:00:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:00:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:00:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:00:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:00:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:00:07' 2024-09-18T00:00:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:00:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:00:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:00:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2175795.2,"bytes_per_snapshot":65273856.0,"last_snapshot_bytes":65273856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"} 2024-09-18T00:00:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:00:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:00:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:00:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:00:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:00:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:00:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:00:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:00:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-18T00:00:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-17 23:59:47' 2024-09-18T00:00:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:00:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-17 23:59:47' 2024-09-18T00:00:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:00:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:00:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:00:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10801c5de6-b3f3-4fca-ba00-b00b417252d2up+replayingreplaying, {"bytes_per_second":2175795.2,"bytes_per_snapshot":65273856.0,"last_snapshot_bytes":65273856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617549,"remote_snapshot_timestamp":1726617549,"replay_state":"idle"}42634266mirror.0smithi1922024-09-17 23:59:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:00:07' 2024-09-18T00:00:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:00:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:00:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-17 23:59:47' 2024-09-18T00:00:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:00:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:00:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:00:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:00:10.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:00:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-18T00:00:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:00:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:10.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-18T00:00:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-18T00:00:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:00:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-18T00:00:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:00:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:00:11.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary10 2024-09-18T00:00:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary10 2024-09-18T00:00:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..10} 2024-09-18T00:00:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS=() 2024-09-18T00:00:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS=() 2024-09-18T00:00:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS=() 2024-09-18T00:00:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS=() 2024-09-18T00:00:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ RET=0 2024-09-18T00:00:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:00:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-18T00:00:11.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary1 2024-09-18T00:00:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-18T00:00:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:00:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:00:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-18T00:00:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:00:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:00:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:00:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:00:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-18T00:00:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:00:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:00:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-18T00:00:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:00:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:12.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-18T00:00:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:00:12.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-18T00:00:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary1': 2024-09-18T00:00:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:00:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:00:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:00:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 22cd72edea16 2024-09-18T00:00:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.22cd72edea16 2024-09-18T00:00:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:00:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:00:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:00:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:00:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:00:12 2024 2024-09-18T00:00:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:00:12 2024 2024-09-18T00:00:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:00:12 2024 2024-09-18T00:00:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:00:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:00:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: d6211d46-bc9d-4a89-aad7-a72121a4f33a 2024-09-18T00:00:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:00:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:00:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:00:12.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-18T00:00:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd0 2024-09-18T00:00:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:00:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-18T00:00:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:00:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:00:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:00:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 4950b7ad-056b-4a5d-a40f-96dbaf7fbdbb 2024-09-18T00:00:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:00:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:00:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:00:12.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:00:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:00:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:00:12.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:12.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary1 2024-09-18T00:00:12.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-18T00:00:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:00:12.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary1 2024-09-18T00:00:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:00:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-18T00:00:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:00:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:00:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary1 2024-09-18T00:00:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-18T00:00:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary2 2024-09-18T00:00:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-18T00:00:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:00:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:00:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary1 2024-09-18T00:00:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-18T00:00:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:00:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-18T00:00:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:00:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:00:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:00:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:00:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:00:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:00:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-18T00:00:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-18T00:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-18T00:00:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:00:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-18T00:00:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:00:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1326 2024-09-18T00:00:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-18T00:00:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:13.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary2': 2024-09-18T00:00:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:00:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:00:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:00:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 22ddd68e2610 2024-09-18T00:00:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.22ddd68e2610 2024-09-18T00:00:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:00:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:00:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:00:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:00:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:00:12 2024 2024-09-18T00:00:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:00:12 2024 2024-09-18T00:00:13.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:00:12 2024 2024-09-18T00:00:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:00:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:00:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: ce67ee59-362f-404b-b1ee-4d54275b799a 2024-09-18T00:00:13.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:00:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:00:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:00:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-18T00:00:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd1 2024-09-18T00:00:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:00:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-18T00:00:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:00:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:00:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:00:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: e0b7ec6c-ad8d-45a2-8b22-eed13ac58522 2024-09-18T00:00:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:00:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:00:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:00:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:00:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:00:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:00:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary2 2024-09-18T00:00:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-18T00:00:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:00:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary2 2024-09-18T00:00:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:00:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-18T00:00:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:00:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:00:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:00:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary2 2024-09-18T00:00:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-18T00:00:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:00:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary3 2024-09-18T00:00:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-18T00:00:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:00:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:00:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary2 2024-09-18T00:00:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:00:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-18T00:00:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:00:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-18T00:00:13.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:00:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:00:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:00:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:00:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-18T00:00:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:00:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:00:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-18T00:00:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:00:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-18T00:00:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:00:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-18T00:00:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:00:14.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1328 2024-09-18T00:00:14.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:14.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-18T00:00:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary3': 2024-09-18T00:00:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:00:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:00:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:00:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 22ece790a47f 2024-09-18T00:00:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.22ece790a47f 2024-09-18T00:00:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:00:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:00:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:00:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:00:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:00:13 2024 2024-09-18T00:00:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:00:13 2024 2024-09-18T00:00:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:00:13 2024 2024-09-18T00:00:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:00:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:00:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: e57ec286-d125-4bae-b4a6-90ac73cbdd80 2024-09-18T00:00:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:00:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:00:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:00:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-18T00:00:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd2 2024-09-18T00:00:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:00:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-18T00:00:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:00:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:00:14.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:00:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: d3669238-7dba-412f-8dd1-bcaab30e5cd9 2024-09-18T00:00:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:00:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:00:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:00:14.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:00:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:00:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:00:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary3 2024-09-18T00:00:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-18T00:00:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:00:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary3 2024-09-18T00:00:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:00:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-18T00:00:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:15.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:00:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:00:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:00:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:00:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-18T00:00:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary4 2024-09-18T00:00:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:00:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-18T00:00:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:00:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary3 2024-09-18T00:00:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:00:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:00:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-18T00:00:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:00:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary3 2024-09-18T00:00:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:00:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:00:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:00:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-18T00:00:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-18T00:00:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:00:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:00:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-18T00:00:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:00:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:15.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-18T00:00:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:00:15.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-18T00:00:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:00:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1330 2024-09-18T00:00:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-18T00:00:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary4': 2024-09-18T00:00:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:00:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:00:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:00:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 22f848e40caa 2024-09-18T00:00:15.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.22f848e40caa 2024-09-18T00:00:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:00:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:00:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:00:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:00:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:00:15 2024 2024-09-18T00:00:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:00:15 2024 2024-09-18T00:00:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:00:15 2024 2024-09-18T00:00:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:00:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:00:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: f249e43d-4066-43ba-86cc-03cf748ba631 2024-09-18T00:00:15.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:00:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:00:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:00:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-18T00:00:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd3 2024-09-18T00:00:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:00:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-18T00:00:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:00:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:00:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:00:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: e6d63a2f-b67b-4e0c-8264-b1fdbb516567 2024-09-18T00:00:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:00:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:00:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:00:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:00:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:00:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:00:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary4 2024-09-18T00:00:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-18T00:00:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:00:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary4 2024-09-18T00:00:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:00:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-18T00:00:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:00:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:00:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:00:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:00:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary4 2024-09-18T00:00:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-18T00:00:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:00:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary5 2024-09-18T00:00:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:00:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-18T00:00:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary4 2024-09-18T00:00:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:00:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-18T00:00:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:00:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-18T00:00:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:00:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:00:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:00:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:00:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-18T00:00:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:00:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:00:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-18T00:00:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:00:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:16.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-18T00:00:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:00:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-18T00:00:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:00:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1332 2024-09-18T00:00:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-18T00:00:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:00:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:00:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:00:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary5': 2024-09-18T00:00:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:00:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:00:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:00:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 2309332fcc1 2024-09-18T00:00:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.2309332fcc1 2024-09-18T00:00:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:00:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:00:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:00:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:00:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:00:16 2024 2024-09-18T00:00:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:00:16 2024 2024-09-18T00:00:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:00:16 2024 2024-09-18T00:00:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:00:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:00:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: c30f6374-0c21-4774-8ad7-8601c14bc59c 2024-09-18T00:00:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:00:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:00:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:00:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-18T00:00:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd4 2024-09-18T00:00:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:00:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-18T00:00:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:00:16.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:00:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:00:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 8c29d252-3d14-4f02-be03-8e232d61205f 2024-09-18T00:00:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:00:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:00:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:00:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:00:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:00:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:00:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary5 2024-09-18T00:00:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-18T00:00:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:00:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary5 2024-09-18T00:00:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:00:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-18T00:00:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:00:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:00:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:00:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary5 2024-09-18T00:00:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-18T00:00:16.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:00:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary6 2024-09-18T00:00:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-18T00:00:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:00:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:00:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary5 2024-09-18T00:00:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:00:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-18T00:00:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:00:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-18T00:00:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:00:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:00:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:00:16.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:00:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-18T00:00:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:00:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:00:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-18T00:00:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:00:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-18T00:00:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:00:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-18T00:00:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:00:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:17.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:00:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:00:17.651 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:17.649+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4dd0 2024-09-18T00:00:17.651 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:17.649+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb5790 2024-09-18T00:00:17.651 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:17.649+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ea270 2024-09-18T00:00:17.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:00:17.654 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:17.652+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:00:17.654 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:17.652+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-18T00:00:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1335 2024-09-18T00:00:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:17.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-18T00:00:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary6': 2024-09-18T00:00:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:00:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:00:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:00:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 231b1a454431 2024-09-18T00:00:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.231b1a454431 2024-09-18T00:00:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:00:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:00:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:00:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:00:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:00:16 2024 2024-09-18T00:00:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:00:16 2024 2024-09-18T00:00:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:00:16 2024 2024-09-18T00:00:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:00:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:00:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: b7dd2433-07bf-4541-b321-29f2ea0fd195 2024-09-18T00:00:17.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:00:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:00:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:00:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-18T00:00:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd5 2024-09-18T00:00:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:00:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-18T00:00:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:00:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:00:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:00:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 4fcf6ed2-c71b-454a-ae02-d575eb998da6 2024-09-18T00:00:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:00:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:00:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:00:18.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:00:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:00:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:00:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary6 2024-09-18T00:00:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-18T00:00:18.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:00:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary6 2024-09-18T00:00:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:00:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-18T00:00:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:00:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:00:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:00:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:00:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary6 2024-09-18T00:00:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-18T00:00:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:00:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary7 2024-09-18T00:00:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:00:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-18T00:00:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary6 2024-09-18T00:00:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-18T00:00:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:00:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-18T00:00:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:00:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:00:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:00:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-18T00:00:18.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:00:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:00:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-18T00:00:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:18.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-18T00:00:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:00:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-18T00:00:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1334 2024-09-18T00:00:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:00:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:00:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:00:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:00:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-18T00:00:18.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1338 2024-09-18T00:00:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary7': 2024-09-18T00:00:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:00:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:00:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:00:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 232ceed5d0e4 2024-09-18T00:00:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.232ceed5d0e4 2024-09-18T00:00:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:00:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:00:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:00:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:00:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:00:18 2024 2024-09-18T00:00:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:00:18 2024 2024-09-18T00:00:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:00:18 2024 2024-09-18T00:00:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:00:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:00:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 30c054d7-0b90-4939-b77c-2b9679b9f2f1 2024-09-18T00:00:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:00:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:00:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:00:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-18T00:00:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd6 2024-09-18T00:00:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:00:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-18T00:00:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:00:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:00:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:00:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 7a1d7ad3-ebb0-4ce2-a45e-8ae19dbbf437 2024-09-18T00:00:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:00:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:00:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:00:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:00:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:00:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:00:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary7 2024-09-18T00:00:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-18T00:00:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:00:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary7 2024-09-18T00:00:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:00:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-18T00:00:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:00:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:00:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary7 2024-09-18T00:00:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:00:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-18T00:00:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary8 2024-09-18T00:00:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-18T00:00:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary7 2024-09-18T00:00:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:00:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-18T00:00:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:00:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-18T00:00:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:19.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:00:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:00:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:00:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-18T00:00:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:00:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:00:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-18T00:00:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:19.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-18T00:00:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:00:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-18T00:00:19.283 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:19.282+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4dd0 2024-09-18T00:00:19.285 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:19.283+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb5790 2024-09-18T00:00:19.285 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:19.283+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ea270 2024-09-18T00:00:19.286 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:19.284+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:00:19.286 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:19.284+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-18T00:00:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1337 2024-09-18T00:00:19.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1341 2024-09-18T00:00:19.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:00:19.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-18T00:00:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:00:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:00:19.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:00:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary8': 2024-09-18T00:00:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:00:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:00:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:00:19.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 233a1b9b8a1 2024-09-18T00:00:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.233a1b9b8a1 2024-09-18T00:00:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:00:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:00:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:00:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:00:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:00:19 2024 2024-09-18T00:00:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:00:19 2024 2024-09-18T00:00:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:00:19 2024 2024-09-18T00:00:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:00:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:00:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 33d0f384-0c58-4667-93a0-214543e5787e 2024-09-18T00:00:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:00:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:00:19.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:00:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-18T00:00:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd7 2024-09-18T00:00:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:00:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-18T00:00:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:00:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:00:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:00:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 9cbcaf30-f5e8-4750-8fa1-5aea918bdb6b 2024-09-18T00:00:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:00:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:00:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:00:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:00:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:00:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:00:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary8 2024-09-18T00:00:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-18T00:00:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:00:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary8 2024-09-18T00:00:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:00:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-18T00:00:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:00:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:00:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-18T00:00:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary9 2024-09-18T00:00:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:00:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary8 2024-09-18T00:00:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-18T00:00:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:00:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:00:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary8 2024-09-18T00:00:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-18T00:00:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-18T00:00:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:00:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:00:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:00:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-18T00:00:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:00:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:00:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-18T00:00:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-18T00:00:20.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:00:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-18T00:00:20.145 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:20.144+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:00:20.145 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:20.144+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-18T00:00:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1340 2024-09-18T00:00:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:00:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:00:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:00:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:20.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1344 2024-09-18T00:00:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:00:20.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-18T00:00:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary9': 2024-09-18T00:00:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:00:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:00:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:00:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 234b95790c8b 2024-09-18T00:00:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.234b95790c8b 2024-09-18T00:00:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:00:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:00:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:00:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:00:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:00:20 2024 2024-09-18T00:00:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:00:20 2024 2024-09-18T00:00:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:00:20 2024 2024-09-18T00:00:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:00:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:00:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9a2080ab-30fe-4915-bc1c-672b7e20a3c2 2024-09-18T00:00:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:00:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:00:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:00:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-18T00:00:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd8 2024-09-18T00:00:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:00:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-18T00:00:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:00:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:00:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:00:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: c596e923-7c47-4a16-98a8-e64585c83ce4 2024-09-18T00:00:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:00:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:00:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:00:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:00:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:00:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:00:21.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary9 2024-09-18T00:00:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-18T00:00:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:00:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary9 2024-09-18T00:00:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:00:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-18T00:00:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:00:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:00:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:00:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-18T00:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary9 2024-09-18T00:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary10 2024-09-18T00:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-18T00:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:21.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary9 2024-09-18T00:00:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:00:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-18T00:00:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-18T00:00:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:00:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:00:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:00:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:00:21.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-18T00:00:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:00:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:00:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:00:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-18T00:00:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:00:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-18T00:00:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:00:21.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-18T00:00:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1343 2024-09-18T00:00:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:00:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:00:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:00:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1347 2024-09-18T00:00:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:00:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:00:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:00:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:00:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-18T00:00:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary10': 2024-09-18T00:00:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:00:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:00:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:00:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 235c4b6ca6cd 2024-09-18T00:00:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.235c4b6ca6cd 2024-09-18T00:00:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:00:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:00:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:00:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:00:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:00:21 2024 2024-09-18T00:00:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:00:21 2024 2024-09-18T00:00:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:00:21 2024 2024-09-18T00:00:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:00:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:00:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3c 2024-09-18T00:00:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:00:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:00:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:00:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-18T00:00:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd9 2024-09-18T00:00:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:00:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-18T00:00:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:00:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:00:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:00:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 420749b5-13e5-4e4a-b891-f9ba028125ce 2024-09-18T00:00:22.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:00:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:00:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:00:22.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:00:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:00:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:00:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:00:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary10 2024-09-18T00:00:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-18T00:00:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:00:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary10 2024-09-18T00:00:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:00:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-18T00:00:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:00:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:00:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary10 2024-09-18T00:00:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:00:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 82956 2024-09-18T00:00:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:00:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary10 2024-09-18T00:00:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-18T00:00:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:00:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-18T00:00:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:00:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:00:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:00:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:00:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:00:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1351 2024-09-18T00:00:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1346 2024-09-18T00:00:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:00:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:00:23.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:00:23.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:00:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:23.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:00:24.027 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:24.026+0000 7f706e974640 -1 reset not still connected to 0x55df1f803520 2024-09-18T00:00:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1352 2024-09-18T00:00:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1353 2024-09-18T00:00:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1349 2024-09-18T00:00:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1350 2024-09-18T00:00:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:00:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:00:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:00:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:00:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:00:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1355 2024-09-18T00:00:25.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1354 2024-09-18T00:00:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:00:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:00:25.836 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:25.834+0000 7f706e974640 -1 reset not still connected to 0x55df1dcd05b0 2024-09-18T00:00:25.836 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:25.834+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4270 2024-09-18T00:00:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:25.834+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4340 2024-09-18T00:00:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:25.834+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb52b0 2024-09-18T00:00:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:25.834+0000 7f706e974640 -1 reset not still connected to 0x55df246572b0 2024-09-18T00:00:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:25.834+0000 7f706e974640 -1 reset not still connected to 0x55df24657380 2024-09-18T00:00:25.837 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:25.834+0000 7f706e974640 -1 reset not still connected to 0x55df2abc3d40 2024-09-18T00:00:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1356 2024-09-18T00:00:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1357 2024-09-18T00:00:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:00:26.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:26.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:26.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:00:26.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:00:26.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:00:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:00:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:00:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:00:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:00:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:00:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:00:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:00:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:00:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:00:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:00:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1359 2024-09-18T00:00:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1360 2024-09-18T00:00:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:27.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:28.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1358 2024-09-18T00:00:28.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:00:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:00:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:00:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1361 2024-09-18T00:00:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:00:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:00:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1363 2024-09-18T00:00:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1362 2024-09-18T00:00:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:00:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:00:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:00:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:00:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:00:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1364 2024-09-18T00:00:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:00:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:00:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:00:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:00:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:00:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:00:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:30.901 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:30.899+0000 7f706e974640 -1 reset not still connected to 0x55df24657380 2024-09-18T00:00:30.903 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:00:30.901+0000 7f706e974640 -1 reset not still connected to 0x55df2abc3d40 2024-09-18T00:00:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:00:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:00:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1365 2024-09-18T00:00:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1366 2024-09-18T00:00:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:00:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:00:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:00:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:00:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:00:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:00:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:00:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:00:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1367 2024-09-18T00:00:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1369 2024-09-18T00:00:32.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1368 2024-09-18T00:00:32.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:32.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:32.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1370 2024-09-18T00:00:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1372 2024-09-18T00:00:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1371 2024-09-18T00:00:33.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:00:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:33.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:00:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:33.972 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:00:33.970+0000 7f6acfc5f640 -1 reset not still connected to 0x555c23351860 2024-09-18T00:00:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1373 2024-09-18T00:00:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:34.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:00:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:00:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:00:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:00:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:00:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:00:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1374 2024-09-18T00:00:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:00:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:00:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:00:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:35.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:00:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:00:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:00:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:00:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:00:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:00:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:00:35.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1376 2024-09-18T00:00:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1375 2024-09-18T00:00:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:00:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:00:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:00:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:00:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:00:36.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:00:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1379 2024-09-18T00:00:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:00:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:00:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1377 2024-09-18T00:00:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1380 2024-09-18T00:00:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1378 2024-09-18T00:00:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:37.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1382 2024-09-18T00:00:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1383 2024-09-18T00:00:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1381 2024-09-18T00:00:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:00:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:00:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:00:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:00:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:00:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:00:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:00:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:00:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:00:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:00:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1384 2024-09-18T00:00:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1385 2024-09-18T00:00:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:00:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:00:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:00:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:00:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:00:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:00:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:00:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:00:41.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1387 2024-09-18T00:00:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1386 2024-09-18T00:00:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:41.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:00:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:00:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:00:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:00:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:00:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:00:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:00:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:00:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1388 2024-09-18T00:00:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1390 2024-09-18T00:00:42.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:43.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1389 2024-09-18T00:00:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1392 2024-09-18T00:00:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1391 2024-09-18T00:00:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1393 2024-09-18T00:00:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:43.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:00:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:43.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:00:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:00:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:00:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:00:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:00:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:00:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:00:44.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:44.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:00:44.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:44.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:44.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:44.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:00:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1395 2024-09-18T00:00:45.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1394 2024-09-18T00:00:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:00:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:00:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:00:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:00:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:00:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1397 2024-09-18T00:00:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:00:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:00:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:00:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:00:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:00:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:00:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:00:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:00:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:00:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:00:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:00:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1399 2024-09-18T00:00:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1398 2024-09-18T00:00:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1396 2024-09-18T00:00:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1401 2024-09-18T00:00:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1403 2024-09-18T00:00:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1402 2024-09-18T00:00:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1400 2024-09-18T00:00:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:00:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:00:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:00:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:00:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:48.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:00:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:00:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:49.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:00:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1404 2024-09-18T00:00:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1405 2024-09-18T00:00:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:00:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:00:50.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:00:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:00:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:00:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:00:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:00:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:00:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:00:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:00:50.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:00:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:00:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:00:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:00:51.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1410 2024-09-18T00:00:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:00:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:51.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:00:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:00:51.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:00:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:00:51.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:52.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1406 2024-09-18T00:00:52.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1409 2024-09-18T00:00:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1407 2024-09-18T00:00:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1413 2024-09-18T00:00:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1412 2024-09-18T00:00:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1408 2024-09-18T00:00:53.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1411 2024-09-18T00:00:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:00:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:00:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:00:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:00:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:00:53.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:53.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:00:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:00:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:00:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1414 2024-09-18T00:00:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:00:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:00:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:00:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:00:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:00:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:00:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:00:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:00:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1415 2024-09-18T00:00:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1416 2024-09-18T00:00:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:00:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:00:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:00:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:00:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:00:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:00:56.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:00:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:00:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:00:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:00:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:00:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:00:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1418 2024-09-18T00:00:56.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1419 2024-09-18T00:00:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1417 2024-09-18T00:00:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1423 2024-09-18T00:00:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1422 2024-09-18T00:00:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1420 2024-09-18T00:00:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1421 2024-09-18T00:00:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:00:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:00:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:00:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:00:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:00:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:00:59.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:00:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:00:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:00:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:00:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:00:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:00:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:00:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:00:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:00:59.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:01:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1424 2024-09-18T00:01:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:01:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:01:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:01:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1425 2024-09-18T00:01:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:01:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:01:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:01:00.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:01:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:01:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:01:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1426 2024-09-18T00:01:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:01:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:01:01.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:01:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:01:01.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:01.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:01.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:01:01.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:01:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:01:01.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:01:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:01:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1429 2024-09-18T00:01:02.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1427 2024-09-18T00:01:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1430 2024-09-18T00:01:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1428 2024-09-18T00:01:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:03.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1431 2024-09-18T00:01:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1433 2024-09-18T00:01:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:01:03.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:01:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:01:03.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:03.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:01:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:01:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:01:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1432 2024-09-18T00:01:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:01:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:01:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:01:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1435 2024-09-18T00:01:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1434 2024-09-18T00:01:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:01:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:01:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:01:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:01:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:01:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:01:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:01:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:01:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:01:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:01:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:01:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1436 2024-09-18T00:01:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1438 2024-09-18T00:01:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:01:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:01:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:01:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:01:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:01:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:01:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1439 2024-09-18T00:01:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:01:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:01:07.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:01:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:01:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:01:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:01:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:01:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:01:08.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:01:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1437 2024-09-18T00:01:08.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1440 2024-09-18T00:01:08.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1442 2024-09-18T00:01:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1441 2024-09-18T00:01:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1443 2024-09-18T00:01:08.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:01:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:01:09.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:01:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:01:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:01:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:01:10.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1445 2024-09-18T00:01:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:01:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:01:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:01:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1447 2024-09-18T00:01:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1446 2024-09-18T00:01:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1444 2024-09-18T00:01:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:01:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:01:11.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:01:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:01:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:01:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:01:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:01:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:01:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:01:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:01:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:01:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:01:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:01:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:01:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:01:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1448 2024-09-18T00:01:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:12.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1449 2024-09-18T00:01:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1450 2024-09-18T00:01:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:01:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:01:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:01:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1451 2024-09-18T00:01:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1452 2024-09-18T00:01:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1453 2024-09-18T00:01:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:13.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:01:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:01:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:01:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:01:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:01:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:01:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:01:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:01:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:01:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1454 2024-09-18T00:01:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1456 2024-09-18T00:01:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:01:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:01:15.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:01:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:01:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:01:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:01:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:01:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:01:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:01:16.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1455 2024-09-18T00:01:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1457 2024-09-18T00:01:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:01:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:01:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:01:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:01:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:01:16.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:01:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:01:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:01:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:01:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1458 2024-09-18T00:01:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1459 2024-09-18T00:01:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1460 2024-09-18T00:01:17.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1463 2024-09-18T00:01:18.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1462 2024-09-18T00:01:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1461 2024-09-18T00:01:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:01:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:01:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:01:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:01:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:01:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:01:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:01:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:01:19.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:01:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:01:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:01:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:01:20.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1464 2024-09-18T00:01:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1465 2024-09-18T00:01:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:01:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:01:20.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:01:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:01:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:01:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:01:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:01:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:01:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:01:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:01:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:21.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:01:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:01:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1466 2024-09-18T00:01:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:01:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:01:21.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:01:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1467 2024-09-18T00:01:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:01:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:01:21.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:01:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1468 2024-09-18T00:01:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1470 2024-09-18T00:01:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:23.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1471 2024-09-18T00:01:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1469 2024-09-18T00:01:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1472 2024-09-18T00:01:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:01:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:01:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:01:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:01:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:01:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:01:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1473 2024-09-18T00:01:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:01:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:01:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:01:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:01:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:01:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:01:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1474 2024-09-18T00:01:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:01:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:01:25.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:01:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:01:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:01:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:01:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1477 2024-09-18T00:01:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1476 2024-09-18T00:01:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1475 2024-09-18T00:01:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:26.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:01:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:01:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:01:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:01:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:26.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:01:26.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:01:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:01:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:01:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:01:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:27.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:01:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:01:27.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:01:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1478 2024-09-18T00:01:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1479 2024-09-18T00:01:27.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:27.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1482 2024-09-18T00:01:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1480 2024-09-18T00:01:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1481 2024-09-18T00:01:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:01:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:01:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:01:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1483 2024-09-18T00:01:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:01:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:01:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:01:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:01:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:01:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:01:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:01:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:01:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:01:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:01:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:01:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:01:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:01:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:01:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:01:30.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1485 2024-09-18T00:01:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:01:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:01:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:01:31.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1484 2024-09-18T00:01:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1487 2024-09-18T00:01:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:01:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:01:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:01:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:01:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:01:31.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:01:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:01:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:01:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:01:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:32.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1490 2024-09-18T00:01:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1488 2024-09-18T00:01:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1489 2024-09-18T00:01:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1486 2024-09-18T00:01:32.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:32.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1493 2024-09-18T00:01:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1491 2024-09-18T00:01:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1492 2024-09-18T00:01:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:33.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:01:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:01:33.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:01:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:01:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:01:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:01:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:01:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:01:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:01:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1494 2024-09-18T00:01:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:35.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1496 2024-09-18T00:01:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:01:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:01:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:01:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:01:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:01:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:01:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:01:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:01:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:01:35.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:01:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:01:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:01:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1495 2024-09-18T00:01:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:01:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:01:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:01:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:01:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:01:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:01:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:01:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:01:36.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:01:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1499 2024-09-18T00:01:36.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1497 2024-09-18T00:01:37.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1498 2024-09-18T00:01:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1500 2024-09-18T00:01:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1503 2024-09-18T00:01:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1501 2024-09-18T00:01:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1502 2024-09-18T00:01:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:01:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:01:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:01:38.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:01:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:01:38.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:01:38.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:38.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:01:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:01:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:01:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:01:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:01:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:01:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1505 2024-09-18T00:01:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1504 2024-09-18T00:01:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:01:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:01:40.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:01:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:01:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:01:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:01:41.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1507 2024-09-18T00:01:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:01:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:01:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:01:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:01:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:01:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:01:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:01:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:01:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:01:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:01:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:01:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:01:42.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1509 2024-09-18T00:01:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1506 2024-09-18T00:01:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1508 2024-09-18T00:01:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:42.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:42.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1511 2024-09-18T00:01:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:01:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:01:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:01:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:01:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:01:43.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:01:43.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1510 2024-09-18T00:01:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1512 2024-09-18T00:01:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1513 2024-09-18T00:01:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:01:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:01:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:01:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1514 2024-09-18T00:01:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:01:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:01:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:01:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:01:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:01:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:01:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:01:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:01:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:01:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:01:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:01:45.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:01:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1516 2024-09-18T00:01:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1515 2024-09-18T00:01:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1519 2024-09-18T00:01:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:01:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:01:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:01:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:01:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:01:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:01:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:01:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:01:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:01:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1517 2024-09-18T00:01:47.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1518 2024-09-18T00:01:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:01:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:01:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:01:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1520 2024-09-18T00:01:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1521 2024-09-18T00:01:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1523 2024-09-18T00:01:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1522 2024-09-18T00:01:49.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:01:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:01:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:01:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:01:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:01:49.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:01:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:01:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:01:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:01:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:01:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:01:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:01:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:01:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:01:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:01:50.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1524 2024-09-18T00:01:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1525 2024-09-18T00:01:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1527 2024-09-18T00:01:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:01:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:01:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:01:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:51.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:51.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:01:51.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:51.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:51.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:01:51.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:01:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:01:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:01:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:01:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:01:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:01:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:01:52.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1529 2024-09-18T00:01:52.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1526 2024-09-18T00:01:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1528 2024-09-18T00:01:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:52.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1533 2024-09-18T00:01:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1531 2024-09-18T00:01:52.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1530 2024-09-18T00:01:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:01:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:01:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:01:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:01:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:01:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:01:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1532 2024-09-18T00:01:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:01:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:01:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:01:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1535 2024-09-18T00:01:55.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:01:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:01:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:01:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:01:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:01:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:01:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:01:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:01:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:01:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:01:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:01:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:01:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:01:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:01:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:01:56.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1536 2024-09-18T00:01:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1534 2024-09-18T00:01:56.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1538 2024-09-18T00:01:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:01:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:01:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:01:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1541 2024-09-18T00:01:57.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1539 2024-09-18T00:01:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1540 2024-09-18T00:01:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1537 2024-09-18T00:01:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:01:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:01:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:01:57.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:57.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1542 2024-09-18T00:01:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1543 2024-09-18T00:01:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:01:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:01:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:01:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:01:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:01:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:01:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:01:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:01:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:01:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:01:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:01:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:01:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:01:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:01:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:01:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:02:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1544 2024-09-18T00:02:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:02:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:02:00.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:02:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:02:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:02:00.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:02:00.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:00.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:00.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:02:00.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:02:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:02:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:02:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:02:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:02:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1545 2024-09-18T00:02:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:02:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:02:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:02:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1546 2024-09-18T00:02:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:02:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:02:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:02:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1547 2024-09-18T00:02:01.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:01.877 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:02:01.875+0000 7fd413089640 -1 reset not still connected to 0x5604a9af1380 2024-09-18T00:02:01.878 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:02:01.876+0000 7fd413089640 -1 reset not still connected to 0x5604a9ed3380 2024-09-18T00:02:01.878 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:02:01.876+0000 7fd413089640 -1 reset not still connected to 0x5604aaeafee0 2024-09-18T00:02:01.878 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:02:01.876+0000 7fd413089640 -1 reset not still connected to 0x5604b3641ad0 2024-09-18T00:02:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1551 2024-09-18T00:02:02.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1549 2024-09-18T00:02:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1548 2024-09-18T00:02:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:02.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:02.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1550 2024-09-18T00:02:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:02:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:02:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:02:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1552 2024-09-18T00:02:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1553 2024-09-18T00:02:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:02:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:02:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:02:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:03.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:02:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:02:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:02:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:02:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:02:04.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:02:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1554 2024-09-18T00:02:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:02:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:02:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:02:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:02:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:02:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:02:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:02:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:02:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:02:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:05.665 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:02:05.663+0000 7f706e974640 -1 reset not still connected to 0x55df34e9c750 2024-09-18T00:02:06.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1557 2024-09-18T00:02:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1556 2024-09-18T00:02:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1555 2024-09-18T00:02:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:02:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:02:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:02:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:02:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:02:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:02:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:02:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:02:06.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:02:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1560 2024-09-18T00:02:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1559 2024-09-18T00:02:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1558 2024-09-18T00:02:07.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:07.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1561 2024-09-18T00:02:07.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1563 2024-09-18T00:02:08.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1562 2024-09-18T00:02:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:08.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:02:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:02:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:02:09.535 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:02:09.533+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1a846000 2024-09-18T00:02:09.535 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:02:09.533+0000 7f9bfab49640 -1 reset not still connected to 0x55cd2ae50d00 2024-09-18T00:02:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:02:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:02:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:02:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:02:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:02:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:02:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:02:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:02:09.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:02:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:02:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:02:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:02:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:02:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:02:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:02:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1566 2024-09-18T00:02:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:02:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:02:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:02:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:02:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:02:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:02:11.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1567 2024-09-18T00:02:11.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1564 2024-09-18T00:02:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:02:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:02:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:02:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1565 2024-09-18T00:02:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:02:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:02:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:02:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1570 2024-09-18T00:02:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1571 2024-09-18T00:02:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1569 2024-09-18T00:02:12.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1568 2024-09-18T00:02:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:12.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1573 2024-09-18T00:02:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1572 2024-09-18T00:02:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:02:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:02:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:02:13.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:02:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:02:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:02:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:02:14.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:02:14.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:02:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:02:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:02:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:02:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1574 2024-09-18T00:02:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1575 2024-09-18T00:02:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:02:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:02:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:02:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:02:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:02:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:02:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:02:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:02:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:02:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:02:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:02:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:02:16.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1576 2024-09-18T00:02:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1577 2024-09-18T00:02:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:02:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:16.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:02:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:02:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:02:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:02:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:02:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1578 2024-09-18T00:02:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1579 2024-09-18T00:02:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1581 2024-09-18T00:02:17.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:02:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:02:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:02:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:02:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:02:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:02:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1582 2024-09-18T00:02:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1580 2024-09-18T00:02:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:18.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:19.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1583 2024-09-18T00:02:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:02:19.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:02:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:02:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:02:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:02:19.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:02:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1584 2024-09-18T00:02:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1585 2024-09-18T00:02:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:20.591 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:02:20.589+0000 7f9bfab49640 -1 reset not still connected to 0x55cd39c351e0 2024-09-18T00:02:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:02:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:02:20.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:02:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:02:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:02:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:02:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:02:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:02:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:02:21.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1587 2024-09-18T00:02:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1586 2024-09-18T00:02:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:02:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:21.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:02:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:02:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:02:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:02:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:02:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:02:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:02:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:02:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1589 2024-09-18T00:02:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1588 2024-09-18T00:02:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1590 2024-09-18T00:02:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:22.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1592 2024-09-18T00:02:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:02:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:02:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:02:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:02:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:02:23.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:02:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1591 2024-09-18T00:02:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:23.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1593 2024-09-18T00:02:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:02:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:02:24.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:02:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:02:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:02:24.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:02:25.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1595 2024-09-18T00:02:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1594 2024-09-18T00:02:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:02:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:02:25.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:02:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:02:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:02:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:02:25.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:02:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:02:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:02:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:02:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:02:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:02:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:02:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:02:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:02:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:02:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:02:26.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:02:27.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1601 2024-09-18T00:02:27.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1599 2024-09-18T00:02:27.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1596 2024-09-18T00:02:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1597 2024-09-18T00:02:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1598 2024-09-18T00:02:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1600 2024-09-18T00:02:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:27.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:02:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:02:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:02:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:02:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:02:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:02:28.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1602 2024-09-18T00:02:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:28.790 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:02:28.788+0000 7fc275b74640 -1 reset not still connected to 0x559cbda835f0 2024-09-18T00:02:28.792 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:02:28.789+0000 7fc275b74640 -1 reset not still connected to 0x559cbebc1110 2024-09-18T00:02:28.794 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:02:28.793+0000 7fc275b74640 -1 reset not still connected to 0x559cbebc1520 2024-09-18T00:02:28.794 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:02:28.793+0000 7fc275b74640 -1 reset not still connected to 0x559cbfdb1040 2024-09-18T00:02:28.799 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:02:28.796+0000 7fc275b74640 -1 reset not still connected to 0x559cc0420f70 2024-09-18T00:02:28.799 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:02:28.796+0000 7fc275b74640 -1 reset not still connected to 0x559cc0421ba0 2024-09-18T00:02:28.799 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:02:28.796+0000 7fc275b74640 -1 reset not still connected to 0x559cc70c0680 2024-09-18T00:02:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1603 2024-09-18T00:02:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1605 2024-09-18T00:02:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1604 2024-09-18T00:02:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:02:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:02:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:02:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:02:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:02:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:02:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:02:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:02:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:02:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:30.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:02:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:02:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:02:30.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:30.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:02:30.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:30.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:30.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:02:30.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:02:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:02:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:02:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:02:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:02:31.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:31.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:31.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:02:31.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:02:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1608 2024-09-18T00:02:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1609 2024-09-18T00:02:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:31.936 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:02:31.934+0000 7fd413089640 -1 reset not still connected to 0x5604b3641ad0 2024-09-18T00:02:31.936 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:02:31.934+0000 7fd413089640 -1 reset not still connected to 0x5604b3641ba0 2024-09-18T00:02:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:02:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:02:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:02:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1610 2024-09-18T00:02:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1611 2024-09-18T00:02:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1606 2024-09-18T00:02:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1613 2024-09-18T00:02:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:02:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:02:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:02:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:02:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:02:33.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:02:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1612 2024-09-18T00:02:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1607 2024-09-18T00:02:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1614 2024-09-18T00:02:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1615 2024-09-18T00:02:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:34.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:02:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:02:34.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:02:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:02:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:34.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:02:34.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:02:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:02:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:02:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:02:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:02:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:02:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:02:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:02:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:02:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:02:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:02:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:02:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:02:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1617 2024-09-18T00:02:36.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1616 2024-09-18T00:02:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:02:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:02:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:02:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:02:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:02:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:02:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:02:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:02:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:02:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:02:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:02:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1618 2024-09-18T00:02:37.689 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:02:37.686+0000 7f706e974640 -1 reset not still connected to 0x55df1d962340 2024-09-18T00:02:37.689 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:02:37.686+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ebad0 2024-09-18T00:02:37.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1621 2024-09-18T00:02:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1619 2024-09-18T00:02:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1620 2024-09-18T00:02:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:37.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1623 2024-09-18T00:02:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1625 2024-09-18T00:02:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1622 2024-09-18T00:02:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1624 2024-09-18T00:02:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:02:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:02:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:02:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:02:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:02:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:02:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:02:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:02:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:02:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:02:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:02:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:02:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:02:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:02:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:02:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:02:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:02:40.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:02:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1626 2024-09-18T00:02:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1627 2024-09-18T00:02:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:02:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:02:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:02:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:02:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 83089 2024-09-18T00:02:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:02:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 83231 2024-09-18T00:02:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:02:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 83364 2024-09-18T00:02:41.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1628 2024-09-18T00:02:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1631 2024-09-18T00:02:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1629 2024-09-18T00:02:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1630 2024-09-18T00:02:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1632 2024-09-18T00:02:43.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:02:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:02:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:02:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:02:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 83516 2024-09-18T00:02:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:02:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:02:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:02:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:02:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:02:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:02:46.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1633 2024-09-18T00:02:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:02:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 83668 2024-09-18T00:02:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:02:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 83821 2024-09-18T00:02:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:02:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 83974 2024-09-18T00:02:46.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:02:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 84127 2024-09-18T00:02:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1634 2024-09-18T00:02:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:02:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:02:49.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 84300 2024-09-18T00:02:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:02:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:02:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 82957 2024-09-18T00:05:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:05:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:05:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:05:12.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 83090 2024-09-18T00:05:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:05:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:05:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:05:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 83232 2024-09-18T00:05:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:05:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:05:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:05:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 83365 2024-09-18T00:05:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:05:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:05:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:05:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 83517 2024-09-18T00:05:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:05:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:05:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:05:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 83669 2024-09-18T00:05:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:05:18.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:05:18.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:05:18.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 83822 2024-09-18T00:05:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:05:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:05:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:05:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 83975 2024-09-18T00:05:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:05:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:05:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:05:20.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 84128 2024-09-18T00:05:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:05:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:05:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:05:21.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 84301 2024-09-18T00:05:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:05:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:05:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:05:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:05:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:05:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:05:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 1 2024-09-18T00:05:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:05:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:05:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd0 2024-09-18T00:05:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 2 2024-09-18T00:05:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary1 2024-09-18T00:05:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary1 2024-09-18T00:05:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:05:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary1 2024-09-18T00:05:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:05:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:05:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd1 2024-09-18T00:05:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 3 2024-09-18T00:05:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary2 2024-09-18T00:05:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary2 2024-09-18T00:05:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:05:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary2 2024-09-18T00:05:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:05:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd2 2024-09-18T00:05:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:05:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary3 2024-09-18T00:05:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 4 2024-09-18T00:05:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary3 2024-09-18T00:05:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:05:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary3 2024-09-18T00:05:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd3 2024-09-18T00:05:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary4 2024-09-18T00:05:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary4 2024-09-18T00:05:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:05:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:05:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary4 2024-09-18T00:05:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:05:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 5 2024-09-18T00:05:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd4 2024-09-18T00:05:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:05:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary5 2024-09-18T00:05:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary5 2024-09-18T00:05:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:05:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:05:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 6 2024-09-18T00:05:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary5 2024-09-18T00:05:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd5 2024-09-18T00:05:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:05:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:05:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 7 2024-09-18T00:05:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary6 2024-09-18T00:05:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary6 2024-09-18T00:05:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:05:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary6 2024-09-18T00:05:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd6 2024-09-18T00:05:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary7 2024-09-18T00:05:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary7 2024-09-18T00:05:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:05:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:05:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:05:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary7 2024-09-18T00:05:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:05:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:05:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:05:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:05:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 91210 2024-09-18T00:05:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 10 2024-09-18T00:05:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd9 2024-09-18T00:05:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary10 2024-09-18T00:05:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary10 2024-09-18T00:05:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:05:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary10 2024-09-18T00:05:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 9 2024-09-18T00:05:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd8 2024-09-18T00:05:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary9 2024-09-18T00:05:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary9 2024-09-18T00:05:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:05:22.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary9 2024-09-18T00:05:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 8 2024-09-18T00:05:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd7 2024-09-18T00:05:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary8 2024-09-18T00:05:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary8 2024-09-18T00:05:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:05:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary8 2024-09-18T00:05:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-18T00:05:22.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:05:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-18T00:05:22.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:05:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-18T00:05:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:05:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-18T00:05:23.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:05:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:05:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-18T00:05:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:05:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-18T00:05:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:05:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-18T00:05:23.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-18T00:05:23.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:05:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-18T00:05:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-18T00:05:23.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:05:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:05:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=94cb47391c937da243bb80c1a50e5947 2024-09-18T00:05:52.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-18T00:05:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-18T00:05:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:05:52.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-18T00:05:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=3d4615fdb399d1f2c8a6d5085893cb89 2024-09-18T00:05:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-18T00:05:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-18T00:05:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:05:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-18T00:05:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=33143f79f338cc304420a3dbc2ce73f9 2024-09-18T00:05:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-18T00:05:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-18T00:05:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:05:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-18T00:05:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=44825859c5bf935a6e550cdc78ff7e5f 2024-09-18T00:05:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-18T00:05:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=20215fc8b62c770ac98bbcb22096fd6e 2024-09-18T00:05:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-18T00:05:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-18T00:05:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:05:53.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-18T00:05:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-18T00:05:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:05:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-18T00:05:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=ed0126dbe262bd569dcd546c2e2da65b 2024-09-18T00:05:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-18T00:05:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=362f14da51efb3818e10465d7164210f 2024-09-18T00:05:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-18T00:05:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-18T00:05:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:05:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-18T00:05:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-18T00:05:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:05:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-18T00:05:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=bea2ad4253e2f4066e93aa30c6c2832d 2024-09-18T00:05:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-18T00:05:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=5d9bb26d9f9d10f6a4ad541faa19a354 2024-09-18T00:05:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-18T00:05:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-18T00:05:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:05:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-18T00:05:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-18T00:05:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:05:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-18T00:05:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=69ce27cf037bd9523a9e31cc8b84c7fc 2024-09-18T00:05:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-18T00:05:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-18T00:05:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:05:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-18T00:05:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:05:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:05:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-18T00:05:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:05:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:54.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:05:54.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-18T00:05:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:05:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:54.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-18T00:05:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:05:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:05:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-18T00:05:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:05:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:05:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:05:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:05:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:05:55.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:05:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-18T00:05:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:05:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-18T00:05:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:05:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-18T00:05:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:05:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-18T00:05:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:05:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:05:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-18T00:05:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:05:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:05:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-18T00:05:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:05:55.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:55.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:55.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:55.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:55.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-18T00:05:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:05:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-18T00:05:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:05:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:05:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-18T00:05:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:05:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-18T00:05:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:05:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: e57ec286-d125-4bae-b4a6-90ac73cbdd80 2024-09-18T00:05:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:05:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:05:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:52 2024-09-18T00:05:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:05:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:55' 2024-09-18T00:05:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:05:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:05:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:05:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 33d0f384-0c58-4667-93a0-214543e5787e 2024-09-18T00:05:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:05:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:05:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:52 2024-09-18T00:05:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:05:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:55' 2024-09-18T00:05:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:05:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:05:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:05:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3c 2024-09-18T00:05:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:05:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:05:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:52 2024-09-18T00:05:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:05:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:55' 2024-09-18T00:05:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:05:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:05:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:05:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c30f6374-0c21-4774-8ad7-8601c14bc59c 2024-09-18T00:05:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:05:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:05:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:52 2024-09-18T00:05:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:05:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1732608.0,"last_snapshot_bytes":1744896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617761,"remote_snapshot_timestamp":1726617761,"replay_state":"idle"} 2024-09-18T00:05:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:47' 2024-09-18T00:05:56.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:05:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:05:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:05:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-18T00:05:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:05:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-18T00:05:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:05:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:05:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-18T00:05:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:05:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:05:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-18T00:05:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:05:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:05:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-18T00:05:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:05:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-18T00:05:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:05:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 30c054d7-0b90-4939-b77c-2b9679b9f2f1 2024-09-18T00:05:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:05:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:05:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:52 2024-09-18T00:05:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:05:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:56' 2024-09-18T00:05:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:05:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:05:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:05:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d6211d46-bc9d-4a89-aad7-a72121a4f33a 2024-09-18T00:05:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:05:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:05:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:52 2024-09-18T00:05:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:05:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1701888.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617757,"remote_snapshot_timestamp":1726617757,"replay_state":"idle"} 2024-09-18T00:05:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:47' 2024-09-18T00:05:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:05:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:05:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-18T00:05:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:05:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:05:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:05:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9a2080ab-30fe-4915-bc1c-672b7e20a3c2 2024-09-18T00:05:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:05:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:05:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:52 2024-09-18T00:05:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:05:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1691648.0,"last_snapshot_bytes":1626112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617765,"remote_snapshot_timestamp":1726617765,"replay_state":"idle"} 2024-09-18T00:05:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:47' 2024-09-18T00:05:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:05:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:05:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:05:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: f249e43d-4066-43ba-86cc-03cf748ba631 2024-09-18T00:05:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:05:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:05:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:52 2024-09-18T00:05:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:05:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:56' 2024-09-18T00:05:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:05:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:05:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:05:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: b7dd2433-07bf-4541-b321-29f2ea0fd195 2024-09-18T00:05:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:05:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:05:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:52 2024-09-18T00:05:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:05:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1695744.0,"last_snapshot_bytes":1646592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617761,"remote_snapshot_timestamp":1726617761,"replay_state":"idle"} 2024-09-18T00:05:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:47' 2024-09-18T00:05:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:05:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:56.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:05:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-18T00:05:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:05:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:05:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:05:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ce67ee59-362f-404b-b1ee-4d54275b799a 2024-09-18T00:05:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:05:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:05:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:05:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:05:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:05:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:05:57.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:05:57.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:57.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:05:57.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:05:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:05:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:05:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:05:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:05:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:05:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:05:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:05:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57' 2024-09-18T00:05:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:05:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:05:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:05:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:05:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:05:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-18T00:05:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:05:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-18T00:05:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:05:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:05:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:05:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: e57ec286-d125-4bae-b4a6-90ac73cbdd80 2024-09-18T00:05:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:05:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:05:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:05:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:05:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:05:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:05:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:05:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:05:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:05:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:05:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:05:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:05:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:05:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:05:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:05:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-18T00:05:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:05:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:05:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-18T00:05:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:05:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:05:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-18T00:05:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:05:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-18T00:05:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:05:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 33d0f384-0c58-4667-93a0-214543e5787e 2024-09-18T00:05:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:05:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:05:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:05:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:05:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:05:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:05:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:05:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:05:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:05:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:05:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:05:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57' 2024-09-18T00:05:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:05:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:05:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:05:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:05:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:05:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:05:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:05:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3c 2024-09-18T00:05:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:05:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:05:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:05:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:05:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:05:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:05:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:05:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:05:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:05:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:05:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-18T00:05:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:05:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:05:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:05:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:05:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:05:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:05:58.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:05:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:05:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:05:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:05:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c30f6374-0c21-4774-8ad7-8601c14bc59c 2024-09-18T00:05:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:05:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:05:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:05:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:05:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:05:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:05:58.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:05:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:05:58.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:05:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:05:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:05:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:05:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:05:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:05:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:05:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57' 2024-09-18T00:05:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:05:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:05:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:05:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:05:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:05:58.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-18T00:05:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:05:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57' 2024-09-18T00:05:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:05:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:05:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:05:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:05:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:05:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-18T00:05:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:05:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57' 2024-09-18T00:05:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:05:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:05:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:05:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:05:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:05:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-18T00:05:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:05:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:58.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-18T00:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:05:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-18T00:05:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:05:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:05:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-18T00:05:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:05:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:05:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-18T00:05:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:05:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-18T00:05:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:05:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 30c054d7-0b90-4939-b77c-2b9679b9f2f1 2024-09-18T00:05:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:05:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:05:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:05:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:05:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:05:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:05:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:05:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:05:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d6211d46-bc9d-4a89-aad7-a72121a4f33a 2024-09-18T00:05:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:05:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:05:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:05:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:05:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-18T00:05:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:05:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:05:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:05:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:05:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:05:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:05:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:05:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:05:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:05:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:05:58.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:05:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:05:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:05:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:05:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:05:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:05:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:05:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:05:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:05:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:05:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:05:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:05:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-18T00:05:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:05:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-18T00:05:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:05:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9a2080ab-30fe-4915-bc1c-672b7e20a3c2 2024-09-18T00:05:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:05:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:05:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:05:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:05:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:05:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:05:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:05:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:05:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:05:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:05:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:05:58.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:05:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:05:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:05:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:05:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:05:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:05:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: f249e43d-4066-43ba-86cc-03cf748ba631 2024-09-18T00:05:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:05:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:05:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:05:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:05:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:05:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:05:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:05:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:05:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:05:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:05:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:05:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:05:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:05:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:05:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:05:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:05:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ce67ee59-362f-404b-b1ee-4d54275b799a 2024-09-18T00:05:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:05:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:05:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:05:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:05:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:05:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:05:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:05:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:05:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:05:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:05:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:05:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:05:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:05:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:05:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:05:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:05:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:05:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57' 2024-09-18T00:05:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:05:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:05:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:05:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:05:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:05:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-18T00:05:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:05:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57' 2024-09-18T00:05:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:05:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:05:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:05:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:05:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:05:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-18T00:05:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:05:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57' 2024-09-18T00:05:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:05:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:05:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:05:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:05:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:05:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-18T00:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57' 2024-09-18T00:05:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:05:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:05:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:05:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:05:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:05:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-18T00:05:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:05:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57' 2024-09-18T00:05:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:05:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:05:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:05:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:05:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:05:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-18T00:05:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:05:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:05:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-18T00:05:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-18T00:05:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:05:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-18T00:05:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:05:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: b7dd2433-07bf-4541-b321-29f2ea0fd195 2024-09-18T00:05:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:05:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:05:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:05:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:05:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:05:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:05:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:05:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:05:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:05:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:05:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:05:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: e57ec286-d125-4bae-b4a6-90ac73cbdd80 2024-09-18T00:05:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:05:59.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:05:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:05:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:05:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:05:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:05:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:05:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:05:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:05:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:05:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:05:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:05:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:59.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:05:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:05:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:05:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:05:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:05:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:05:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:05:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:05:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:05:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-18T00:05:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:05:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-18T00:05:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-18T00:05:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:05:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-18T00:05:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-18T00:05:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:05:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-18T00:05:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:05:57cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:05:57' 2024-09-18T00:05:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:05:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:05:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:05:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:05:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:05:59.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-18T00:05:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:05:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:05:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:05:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:05:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:05:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57' 2024-09-18T00:05:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:05:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:05:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:05:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:05:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:05:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-18T00:05:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:05:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:05:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-18T00:05:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:05:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 33d0f384-0c58-4667-93a0-214543e5787e 2024-09-18T00:05:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:05:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:05:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:05:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:05:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:05:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:05:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:05:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:05:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:05:59.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:05:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:05:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:05:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:05:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:05:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:05:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:59.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:05:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:05:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3c 2024-09-18T00:05:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:05:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:59.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:05:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:05:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:05:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:05:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:05:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:05:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:05:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:05:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c30f6374-0c21-4774-8ad7-8601c14bc59c 2024-09-18T00:05:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:05:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:05:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:05:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:05:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:05:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:05:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:05:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:05:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:05:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:05:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:05:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:05:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:05:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:05:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:05:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:05:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:05:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:05:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:05:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:05:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:05:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:05:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:05:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:05:59.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:05:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:05:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:05:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:05:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:05:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:05:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:05:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:05:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57' 2024-09-18T00:05:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:05:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:05:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:05:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:05:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:05:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-18T00:05:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:05:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:05:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-18T00:05:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57' 2024-09-18T00:05:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:05:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:05:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57' 2024-09-18T00:05:59.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:05:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:05:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:05:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:05:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:05:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:05:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:05:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:05:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:05:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:05:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-18T00:05:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:05:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:05:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-18T00:05:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:05:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:05:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-18T00:05:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:05:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:05:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:05:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:05:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-18T00:05:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:05:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:05:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:05:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:05:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-18T00:05:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:05:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:05:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-18T00:05:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd0 2024-09-18T00:05:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-18T00:05:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:05:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd1 2024-09-18T00:05:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-18T00:05:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:06:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-18T00:06:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:06:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:06:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-18T00:06:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-18T00:06:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:06:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:06:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:00.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-18T00:06:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:06:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 30c054d7-0b90-4939-b77c-2b9679b9f2f1 2024-09-18T00:06:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:06:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:06:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:06:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:06:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:06:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:06:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:06:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:06:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:06:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:06:00.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:06:00.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:06:00.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:06:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:06:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:06:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:06:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:06:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:06:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:06:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:06:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:06:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:06:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:06:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:06:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:06:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:06:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:06:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-18T00:06:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:06:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:06:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-18T00:06:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-18T00:06:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:06:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:06:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-18T00:06:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:06:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d6211d46-bc9d-4a89-aad7-a72121a4f33a 2024-09-18T00:06:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:06:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:06:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:06:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:06:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:06:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:06:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:06:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:06:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:06:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:06:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:06:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:06:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:06:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:06:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:06:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:06:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:06:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:06:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:06:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:06:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:06:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:06:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:06:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:06:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:06:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:06:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:00.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:06:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:06:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9a2080ab-30fe-4915-bc1c-672b7e20a3c2 2024-09-18T00:06:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:06:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:06:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:06:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:06:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:06:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:06:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:06:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:06:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:06:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:06:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:06:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:06:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:06:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:06:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:06:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:06:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:06:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:06:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:06:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:06:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:06:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:06:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:06:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:06:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:06:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:06:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:06:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:06:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: f249e43d-4066-43ba-86cc-03cf748ba631 2024-09-18T00:06:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:06:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:06:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:06:00.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:06:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:06:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:06:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:06:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:06:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:06:00.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:06:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:06:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:06:00.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:06:00.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:06:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:06:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:06:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:06:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:06:00.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:06:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:06:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:06:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:06:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:06:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:06:00.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:06:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:06:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:00.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:06:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-18T00:06:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:06:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:06:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:00.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-18T00:06:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57' 2024-09-18T00:06:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:06:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:06:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:06:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:06:00.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:06:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:06:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:06:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:06:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-18T00:06:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:06:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:06:00.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-18T00:06:00.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57' 2024-09-18T00:06:00.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:06:00.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:06:00.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:06:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:06:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:06:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:06:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:06:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:06:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-18T00:06:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:06:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:06:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-18T00:06:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:06:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: b7dd2433-07bf-4541-b321-29f2ea0fd195 2024-09-18T00:06:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:06:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:06:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:06:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57 2024-09-18T00:06:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:06:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:06:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:06:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:06:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:05:57' 2024-09-18T00:06:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:06:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:06:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:06:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:06:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:06:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:06:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:06:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:06:00.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:06:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:06:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:06:00.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:06:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:06:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-18 00:05:57' 2024-09-18T00:06:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:06:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:06:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:05:57' 2024-09-18T00:06:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:06:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57' 2024-09-18T00:06:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:06:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:06:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:06:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:06:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:06:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:06:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:06:00.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:06:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-18T00:06:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:06:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:06:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-18T00:06:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:06:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:06:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57' 2024-09-18T00:06:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:06:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:06:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:06:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:06:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:06:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:06:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:06:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary833d0f384-0c58-4667-93a0-214543e5787eup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:05:57cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:05:57' 2024-09-18T00:06:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:06:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:06:00.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:06:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:06:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:06:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-18T00:06:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:06:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:06:00.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-18T00:06:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:05:57' 2024-09-18T00:06:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:06:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:06:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:06:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:06:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:06:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:06:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-18T00:06:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:06:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:06:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-18T00:06:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:06:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:06:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-18T00:06:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:06:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:06:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-18T00:06:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:06:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:06:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-18T00:06:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:06:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:06:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-18T00:06:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:06:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:06:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-18T00:06:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd3 2024-09-18T00:06:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:06:00.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-18T00:06:00.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd2 2024-09-18T00:06:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-18T00:06:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd4 2024-09-18T00:06:00.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:06:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:06:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-18T00:06:00.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd5 2024-09-18T00:06:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-18T00:06:00.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:06:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd6 2024-09-18T00:06:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:06:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-18T00:06:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:06:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:06:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:06:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:06:01.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:06:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:06:01.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:06:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:06:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:06:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-18T00:06:01.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-18T00:06:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-18T00:06:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd7 2024-09-18T00:06:01.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-18T00:06:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd8 2024-09-18T00:06:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:06:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:06:01.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-18T00:06:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd9 2024-09-18T00:06:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-18T00:06:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:06:02.762 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:06:02.759+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:06:02.762 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:06:02.759+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-18T00:06:02.763 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:06:02.759+0000 7f706e974640 -1 reset not still connected to 0x55df2670dd40 2024-09-18T00:06:02.763 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:06:02.759+0000 7f706e974640 -1 reset not still connected to 0x55df2e30dd40 2024-09-18T00:06:02.763 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:06:02.759+0000 7f706e974640 -1 reset not still connected to 0x55df2f6bcdd0 2024-09-18T00:06:02.763 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:06:02.759+0000 7f706e974640 -1 reset not still connected to 0x55df2f83a0d0 2024-09-18T00:06:07.418 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:06:07.415+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4270 2024-09-18T00:06:07.420 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:06:07.418+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4340 2024-09-18T00:06:07.420 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:06:07.418+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb52b0 2024-09-18T00:06:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=94cb47391c937da243bb80c1a50e5947 2024-09-18T00:06:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-18T00:06:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 94cb47391c937da243bb80c1a50e5947 != \9\4\c\b\4\7\3\9\1\c\9\3\7\d\a\2\4\3\b\b\8\0\c\1\a\5\0\e\5\9\4\7 ]] 2024-09-18T00:06:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=69ce27cf037bd9523a9e31cc8b84c7fc 2024-09-18T00:06:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-18T00:06:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 69ce27cf037bd9523a9e31cc8b84c7fc != \6\9\c\e\2\7\c\f\0\3\7\b\d\9\5\2\3\a\9\e\3\1\c\c\8\b\8\4\c\7\f\c ]] 2024-09-18T00:06:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=20215fc8b62c770ac98bbcb22096fd6e 2024-09-18T00:06:31.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-18T00:06:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 20215fc8b62c770ac98bbcb22096fd6e != \2\0\2\1\5\f\c\8\b\6\2\c\7\7\0\a\c\9\8\b\b\c\b\2\2\0\9\6\f\d\6\e ]] 2024-09-18T00:06:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:06:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 91211 2024-09-18T00:06:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:06:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 91212 2024-09-18T00:06:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:06:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 91214 2024-09-18T00:06:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=44825859c5bf935a6e550cdc78ff7e5f 2024-09-18T00:06:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-18T00:06:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 44825859c5bf935a6e550cdc78ff7e5f != \4\4\8\2\5\8\5\9\c\5\b\f\9\3\5\a\6\e\5\5\0\c\d\c\7\8\f\f\7\e\5\f ]] 2024-09-18T00:06:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=ed0126dbe262bd569dcd546c2e2da65b 2024-09-18T00:06:32.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-18T00:06:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ ed0126dbe262bd569dcd546c2e2da65b != \e\d\0\1\2\6\d\b\e\2\6\2\b\d\5\6\9\d\c\d\5\4\6\c\2\e\2\d\a\6\5\b ]] 2024-09-18T00:06:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:06:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 91216 2024-09-18T00:06:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=3d4615fdb399d1f2c8a6d5085893cb89 2024-09-18T00:06:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-18T00:06:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=bea2ad4253e2f4066e93aa30c6c2832d 2024-09-18T00:06:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-18T00:06:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=33143f79f338cc304420a3dbc2ce73f9 2024-09-18T00:06:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-18T00:06:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 3d4615fdb399d1f2c8a6d5085893cb89 != \3\d\4\6\1\5\f\d\b\3\9\9\d\1\f\2\c\8\a\6\d\5\0\8\5\8\9\3\c\b\8\9 ]] 2024-09-18T00:06:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ bea2ad4253e2f4066e93aa30c6c2832d != \b\e\a\2\a\d\4\2\5\3\e\2\f\4\0\6\6\e\9\3\a\a\3\0\c\6\c\2\8\3\2\d ]] 2024-09-18T00:06:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:06:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 91219 2024-09-18T00:06:33.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 33143f79f338cc304420a3dbc2ce73f9 != \3\3\1\4\3\f\7\9\f\3\3\8\c\c\3\0\4\4\2\0\a\3\d\b\c\2\c\e\7\3\f\9 ]] 2024-09-18T00:06:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=5d9bb26d9f9d10f6a4ad541faa19a354 2024-09-18T00:06:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-18T00:06:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 5d9bb26d9f9d10f6a4ad541faa19a354 != \5\d\9\b\b\2\6\d\9\f\9\d\1\0\f\6\a\4\a\d\5\4\1\f\a\a\1\9\a\3\5\4 ]] 2024-09-18T00:06:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=362f14da51efb3818e10465d7164210f 2024-09-18T00:06:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-18T00:06:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 362f14da51efb3818e10465d7164210f != \3\6\2\f\1\4\d\a\5\1\e\f\b\3\8\1\8\e\1\0\4\6\5\d\7\1\6\4\2\1\0\f ]] 2024-09-18T00:06:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:06:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 91220 2024-09-18T00:06:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:06:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 91223 2024-09-18T00:06:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:06:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 91224 2024-09-18T00:06:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:06:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 91226 2024-09-18T00:06:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:06:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:06:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-18T00:06:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-18T00:06:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:06:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:06:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-18T00:06:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:06:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:06:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-18T00:06:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:06:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:34.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:06:34.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d6211d46-bc9d-4a89-aad7-a72121a4f33a 2024-09-18T00:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":7109017.6,"bytes_per_snapshot":71090176.0,"last_snapshot_bytes":71090176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"} 2024-09-18T00:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:17 2024-09-18T00:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:32' 2024-09-18T00:06:34.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:06:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:06:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":7109017.6,"bytes_per_snapshot":71090176.0,"last_snapshot_bytes":71090176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"} 2024-09-18T00:06:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:06:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:06:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:06:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:06:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:06:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:06:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:17' 2024-09-18T00:06:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-18 00:06:17' 2024-09-18T00:06:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:06:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:06:17' 2024-09-18T00:06:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:34.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:06:34.629 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:06:34.628+0000 7f9bfab49640 -1 reset not still connected to 0x55cd23451a00 2024-09-18T00:06:34.630 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:06:34.628+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3374f930 2024-09-18T00:06:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1d6211d46-bc9d-4a89-aad7-a72121a4f33aup+replayingreplaying, {"bytes_per_second":7109017.6,"bytes_per_snapshot":71090176.0,"last_snapshot_bytes":71090176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:32image-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+replayingreplaying, {"bytes_per_second":7035289.6,"bytes_per_snapshot":70352896.0,"last_snapshot_bytes":70352896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:32image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+replayingreplaying, {"bytes_per_second":6676889.6,"bytes_per_snapshot":66768896.0,"last_snapshot_bytes":66768896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617959,"remote_snapshot_timestamp":1726617959,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:32image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+replayingreplaying, {"bytes_per_second":7155712.0,"bytes_per_snapshot":71557120.0,"last_snapshot_bytes":71557120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617959,"remote_snapshot_timestamp":1726617959,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:32image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+replayingreplaying, {"bytes_per_second":7190118.4,"bytes_per_snapshot":71901184.0,"last_snapshot_bytes":71901184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:32image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+replayingreplaying, {"bytes_per_second":7181926.4,"bytes_per_snapshot":71819264.0,"last_snapshot_bytes":71819264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:32image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+replayingreplaying, {"bytes_per_second":7201587.2,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:32image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+replayingreplaying, {"bytes_per_second":7215104.0,"bytes_per_snapshot":72151040.0,"last_snapshot_bytes":72151040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:32image-primary833d0f384-0c58-4667-93a0-214543e5787eup+replayingreplaying, {"bytes_per_second":7210188.8,"bytes_per_snapshot":72101888.0,"last_snapshot_bytes":72101888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:32image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+replayingreplaying, {"bytes_per_second":7186432.0,"bytes_per_snapshot":71864320.0,"last_snapshot_bytes":71864320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:32' 2024-09-18T00:06:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:06:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:06:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:17' 2024-09-18T00:06:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:06:34.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:06:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:06:34.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-18T00:06:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:34.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:06:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-18T00:06:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-18T00:06:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:06:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-18T00:06:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:06:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:06:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary1 2024-09-18T00:06:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary1 2024-09-18T00:06:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:06:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-18T00:06:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-18T00:06:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:06:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:06:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-18T00:06:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:06:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:06:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-18T00:06:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:06:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:06:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:06:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ce67ee59-362f-404b-b1ee-4d54275b799a 2024-09-18T00:06:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:06:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":6676889.6,"bytes_per_snapshot":66768896.0,"last_snapshot_bytes":66768896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617959,"remote_snapshot_timestamp":1726617959,"replay_state":"idle"} 2024-09-18T00:06:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:06:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:17 2024-09-18T00:06:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:06:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:06:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:06:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:06:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:37' 2024-09-18T00:06:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:06:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:06:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:06:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":6676889.6,"bytes_per_snapshot":66768896.0,"last_snapshot_bytes":66768896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617959,"remote_snapshot_timestamp":1726617959,"replay_state":"idle"} 2024-09-18T00:06:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:06:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:06:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:37.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:06:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:06:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:06:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:06:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:17' 2024-09-18T00:06:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-18 00:06:17' 2024-09-18T00:06:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:06:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:06:17' 2024-09-18T00:06:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:06:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK942634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+replayingreplaying, {"bytes_per_second":7035289.6,"bytes_per_snapshot":70352896.0,"last_snapshot_bytes":70352896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37image-primary2ce67ee59-362f-404b-b1ee-4d54275b799aup+replayingreplaying, {"bytes_per_second":6676889.6,"bytes_per_snapshot":66768896.0,"last_snapshot_bytes":66768896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617959,"remote_snapshot_timestamp":1726617959,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+replayingreplaying, {"bytes_per_second":7155712.0,"bytes_per_snapshot":71557120.0,"last_snapshot_bytes":71557120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617959,"remote_snapshot_timestamp":1726617959,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+replayingreplaying, {"bytes_per_second":7190118.4,"bytes_per_snapshot":71901184.0,"last_snapshot_bytes":71901184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+replayingreplaying, {"bytes_per_second":7181926.4,"bytes_per_snapshot":71819264.0,"last_snapshot_bytes":71819264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+replayingreplaying, {"bytes_per_second":7201587.2,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+replayingreplaying, {"bytes_per_second":7215104.0,"bytes_per_snapshot":72151040.0,"last_snapshot_bytes":72151040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37image-primary833d0f384-0c58-4667-93a0-214543e5787eup+replayingreplaying, {"bytes_per_second":7210188.8,"bytes_per_snapshot":72101888.0,"last_snapshot_bytes":72101888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+replayingreplaying, {"bytes_per_second":7186432.0,"bytes_per_snapshot":71864320.0,"last_snapshot_bytes":71864320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37' 2024-09-18T00:06:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:06:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:06:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:17' 2024-09-18T00:06:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:06:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:06:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:06:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-18T00:06:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:06:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-18T00:06:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-18T00:06:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:06:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-18T00:06:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:06:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:06:39.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary2 2024-09-18T00:06:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary2 2024-09-18T00:06:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:06:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-18T00:06:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-18T00:06:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:06:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:06:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-18T00:06:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:06:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:06:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-18T00:06:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:06:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:06:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:06:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: e57ec286-d125-4bae-b4a6-90ac73cbdd80 2024-09-18T00:06:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:06:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":7155712.0,"bytes_per_snapshot":71557120.0,"last_snapshot_bytes":71557120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617959,"remote_snapshot_timestamp":1726617959,"replay_state":"idle"} 2024-09-18T00:06:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:06:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:17 2024-09-18T00:06:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:06:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:06:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:06:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:06:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:37' 2024-09-18T00:06:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:06:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:06:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:06:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":7155712.0,"bytes_per_snapshot":71557120.0,"last_snapshot_bytes":71557120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617959,"remote_snapshot_timestamp":1726617959,"replay_state":"idle"} 2024-09-18T00:06:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:06:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:06:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:06:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:06:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:06:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:06:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:17' 2024-09-18T00:06:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-18 00:06:17' 2024-09-18T00:06:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:06:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:06:17' 2024-09-18T00:06:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:06:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK842634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+replayingreplaying, {"bytes_per_second":7035289.6,"bytes_per_snapshot":70352896.0,"last_snapshot_bytes":70352896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37image-primary3e57ec286-d125-4bae-b4a6-90ac73cbdd80up+replayingreplaying, {"bytes_per_second":7155712.0,"bytes_per_snapshot":71557120.0,"last_snapshot_bytes":71557120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617959,"remote_snapshot_timestamp":1726617959,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+replayingreplaying, {"bytes_per_second":7190118.4,"bytes_per_snapshot":71901184.0,"last_snapshot_bytes":71901184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+replayingreplaying, {"bytes_per_second":7181926.4,"bytes_per_snapshot":71819264.0,"last_snapshot_bytes":71819264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+replayingreplaying, {"bytes_per_second":7201587.2,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+replayingreplaying, {"bytes_per_second":7215104.0,"bytes_per_snapshot":72151040.0,"last_snapshot_bytes":72151040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37image-primary833d0f384-0c58-4667-93a0-214543e5787eup+replayingreplaying, {"bytes_per_second":7210188.8,"bytes_per_snapshot":72101888.0,"last_snapshot_bytes":72101888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+replayingreplaying, {"bytes_per_second":7186432.0,"bytes_per_snapshot":71864320.0,"last_snapshot_bytes":71864320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:37' 2024-09-18T00:06:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:06:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:06:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:17' 2024-09-18T00:06:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:06:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:06:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:06:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-18T00:06:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:06:40.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-18T00:06:40.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-18T00:06:42.409 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:06:42.408+0000 7f0114255640 -1 reset not still connected to 0x557d0a065ee0 2024-09-18T00:06:42.410 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:06:42.409+0000 7f0114255640 -1 reset not still connected to 0x557d0b464750 2024-09-18T00:06:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:06:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-18T00:06:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:06:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:06:42.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary3 2024-09-18T00:06:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary3 2024-09-18T00:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-18T00:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-18T00:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-18T00:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:06:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:06:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-18T00:06:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:06:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-18T00:06:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:06:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: f249e43d-4066-43ba-86cc-03cf748ba631 2024-09-18T00:06:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:06:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":7190118.4,"bytes_per_snapshot":71901184.0,"last_snapshot_bytes":71901184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"} 2024-09-18T00:06:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:06:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:17 2024-09-18T00:06:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:06:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:06:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:06:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:06:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:42' 2024-09-18T00:06:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:06:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:06:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:06:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":7190118.4,"bytes_per_snapshot":71901184.0,"last_snapshot_bytes":71901184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"} 2024-09-18T00:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:06:43.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:06:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:06:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:06:43.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:06:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:17' 2024-09-18T00:06:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-18 00:06:17' 2024-09-18T00:06:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:43.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:06:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:06:17' 2024-09-18T00:06:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:06:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK742634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+replayingreplaying, {"bytes_per_second":7035289.6,"bytes_per_snapshot":70352896.0,"last_snapshot_bytes":70352896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:42image-primary4f249e43d-4066-43ba-86cc-03cf748ba631up+replayingreplaying, {"bytes_per_second":7190118.4,"bytes_per_snapshot":71901184.0,"last_snapshot_bytes":71901184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:42image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+replayingreplaying, {"bytes_per_second":7181926.4,"bytes_per_snapshot":71819264.0,"last_snapshot_bytes":71819264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:42image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+replayingreplaying, {"bytes_per_second":7201587.2,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:42image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+replayingreplaying, {"bytes_per_second":7215104.0,"bytes_per_snapshot":72151040.0,"last_snapshot_bytes":72151040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:42image-primary833d0f384-0c58-4667-93a0-214543e5787eup+replayingreplaying, {"bytes_per_second":7210188.8,"bytes_per_snapshot":72101888.0,"last_snapshot_bytes":72101888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:42image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+replayingreplaying, {"bytes_per_second":7186432.0,"bytes_per_snapshot":71864320.0,"last_snapshot_bytes":71864320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:42' 2024-09-18T00:06:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:06:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:06:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:17' 2024-09-18T00:06:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:06:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:06:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:06:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:43.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-18T00:06:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:06:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-18T00:06:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-18T00:06:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:06:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-18T00:06:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:44.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:06:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:06:44.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary4 2024-09-18T00:06:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary4 2024-09-18T00:06:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:06:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-18T00:06:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-18T00:06:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:06:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:06:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-18T00:06:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:06:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:06:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-18T00:06:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:06:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-18T00:06:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:06:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c30f6374-0c21-4774-8ad7-8601c14bc59c 2024-09-18T00:06:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:06:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":7181926.4,"bytes_per_snapshot":71819264.0,"last_snapshot_bytes":71819264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"} 2024-09-18T00:06:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:06:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:17 2024-09-18T00:06:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:06:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:06:46.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:06:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:06:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:42' 2024-09-18T00:06:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:06:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:06:46.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:06:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":7181926.4,"bytes_per_snapshot":71819264.0,"last_snapshot_bytes":71819264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"} 2024-09-18T00:06:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:06:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:06:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:06:46.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:06:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:06:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:06:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:17' 2024-09-18T00:06:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-18 00:06:17' 2024-09-18T00:06:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:06:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:06:17' 2024-09-18T00:06:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:06:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK642634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+replayingreplaying, {"bytes_per_second":7035289.6,"bytes_per_snapshot":70352896.0,"last_snapshot_bytes":70352896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:42image-primary5c30f6374-0c21-4774-8ad7-8601c14bc59cup+replayingreplaying, {"bytes_per_second":7181926.4,"bytes_per_snapshot":71819264.0,"last_snapshot_bytes":71819264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:42image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+replayingreplaying, {"bytes_per_second":7201587.2,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:42image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+replayingreplaying, {"bytes_per_second":7215104.0,"bytes_per_snapshot":72151040.0,"last_snapshot_bytes":72151040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:42image-primary833d0f384-0c58-4667-93a0-214543e5787eup+replayingreplaying, {"bytes_per_second":7210188.8,"bytes_per_snapshot":72101888.0,"last_snapshot_bytes":72101888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:42image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+replayingreplaying, {"bytes_per_second":7186432.0,"bytes_per_snapshot":71864320.0,"last_snapshot_bytes":71864320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:42' 2024-09-18T00:06:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:06:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:06:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:17' 2024-09-18T00:06:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:06:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:06:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:46.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:06:46.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-18T00:06:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:06:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-18T00:06:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-18T00:06:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:06:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-18T00:06:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:06:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:06:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary5 2024-09-18T00:06:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary5 2024-09-18T00:06:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:06:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-18T00:06:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-18T00:06:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:06:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:06:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-18T00:06:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:06:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:47.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:06:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-18T00:06:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:06:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:06:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:06:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: b7dd2433-07bf-4541-b321-29f2ea0fd195 2024-09-18T00:06:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:06:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"} 2024-09-18T00:06:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:06:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:47 2024-09-18T00:06:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:06:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:06:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:06:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:06:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:47' 2024-09-18T00:06:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:06:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:06:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:06:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"} 2024-09-18T00:06:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:06:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:06:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:06:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:06:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:06:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:06:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:47' 2024-09-18T00:06:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-18 00:06:47' 2024-09-18T00:06:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:06:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:06:47' 2024-09-18T00:06:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:06:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK542634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70352896.0,"last_snapshot_bytes":70352896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:47image-primary6b7dd2433-07bf-4541-b321-29f2ea0fd195up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72015872.0,"last_snapshot_bytes":72015872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:47image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72151040.0,"last_snapshot_bytes":72151040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:47image-primary833d0f384-0c58-4667-93a0-214543e5787eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72101888.0,"last_snapshot_bytes":72101888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:47image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71864320.0,"last_snapshot_bytes":71864320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:47' 2024-09-18T00:06:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:06:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:06:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:47' 2024-09-18T00:06:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:06:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:06:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:48.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:06:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-18T00:06:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:06:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-18T00:06:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-18T00:06:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:06:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-18T00:06:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:06:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:06:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary6 2024-09-18T00:06:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary6 2024-09-18T00:06:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:06:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-18T00:06:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-18T00:06:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:06:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:06:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-18T00:06:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:06:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:06:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-18T00:06:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:06:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:06:51.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:06:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 30c054d7-0b90-4939-b77c-2b9679b9f2f1 2024-09-18T00:06:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:06:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72151040.0,"last_snapshot_bytes":72151040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"} 2024-09-18T00:06:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:06:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:47 2024-09-18T00:06:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:06:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:06:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:06:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:06:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:47' 2024-09-18T00:06:51.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:06:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:06:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:06:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72151040.0,"last_snapshot_bytes":72151040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"} 2024-09-18T00:06:51.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:06:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:06:51.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:06:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:06:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:06:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:06:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:47' 2024-09-18T00:06:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-18 00:06:47' 2024-09-18T00:06:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:06:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:06:47' 2024-09-18T00:06:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:06:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70352896.0,"last_snapshot_bytes":70352896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:47image-primary730c054d7-0b90-4939-b77c-2b9679b9f2f1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72151040.0,"last_snapshot_bytes":72151040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:47image-primary833d0f384-0c58-4667-93a0-214543e5787eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72101888.0,"last_snapshot_bytes":72101888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:47image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71864320.0,"last_snapshot_bytes":71864320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:47' 2024-09-18T00:06:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:06:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:06:51.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:47' 2024-09-18T00:06:51.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:06:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:06:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:06:51.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-18T00:06:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:06:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-18T00:06:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-18T00:06:52.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:06:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-18T00:06:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:06:52.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:06:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary7 2024-09-18T00:06:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary7 2024-09-18T00:06:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:06:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-18T00:06:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-18T00:06:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-18T00:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:52.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:06:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-18T00:06:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:06:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:06:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:06:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 33d0f384-0c58-4667-93a0-214543e5787e 2024-09-18T00:06:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:06:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72101888.0,"last_snapshot_bytes":72101888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"} 2024-09-18T00:06:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:06:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:47 2024-09-18T00:06:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:06:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:06:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:06:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:06:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:52' 2024-09-18T00:06:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:06:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:06:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:06:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72101888.0,"last_snapshot_bytes":72101888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"} 2024-09-18T00:06:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:06:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:06:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:06:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:06:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:06:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:06:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:47' 2024-09-18T00:06:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-18 00:06:47' 2024-09-18T00:06:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:06:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:06:47' 2024-09-18T00:06:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:06:54.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK342634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70352896.0,"last_snapshot_bytes":70352896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:52image-primary833d0f384-0c58-4667-93a0-214543e5787eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72101888.0,"last_snapshot_bytes":72101888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:52image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71864320.0,"last_snapshot_bytes":71864320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:52' 2024-09-18T00:06:54.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:06:54.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:06:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:47' 2024-09-18T00:06:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:06:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:06:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:06:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-18T00:06:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:06:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-18T00:06:54.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-18T00:06:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:06:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-18T00:06:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:06:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:06:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary8 2024-09-18T00:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary8 2024-09-18T00:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-18T00:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-18T00:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-18T00:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:06:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-18T00:06:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:06:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:56.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:06:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:06:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9a2080ab-30fe-4915-bc1c-672b7e20a3c2 2024-09-18T00:06:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:06:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71864320.0,"last_snapshot_bytes":71864320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"} 2024-09-18T00:06:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:06:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:47 2024-09-18T00:06:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:06:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:06:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:06:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:06:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:52' 2024-09-18T00:06:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:06:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:06:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:06:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71864320.0,"last_snapshot_bytes":71864320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"} 2024-09-18T00:06:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:06:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:06:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:06:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:06:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:06:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:06:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:47' 2024-09-18T00:06:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-18 00:06:47' 2024-09-18T00:06:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:06:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:06:47' 2024-09-18T00:06:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:06:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK242634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70352896.0,"last_snapshot_bytes":70352896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:52image-primary99a2080ab-30fe-4915-bc1c-672b7e20a3c2up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71864320.0,"last_snapshot_bytes":71864320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617961,"remote_snapshot_timestamp":1726617961,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:52' 2024-09-18T00:06:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:06:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:06:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:47' 2024-09-18T00:06:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:06:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:06:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:06:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:06:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:06:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-18T00:06:57.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:06:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:06:57.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-18T00:06:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-18T00:06:58.576 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:06:58.571+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ea270 2024-09-18T00:06:58.581 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:06:58.578+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:06:58.582 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:06:58.580+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-18T00:06:58.582 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:06:58.580+0000 7f706e974640 -1 reset not still connected to 0x55df2670dd40 2024-09-18T00:06:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:06:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-18T00:06:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:06:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:06:58.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary9 2024-09-18T00:06:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:06:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary9 2024-09-18T00:06:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:06:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-18T00:06:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-18T00:06:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:06:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:06:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-18T00:06:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:06:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:06:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:06:59.052 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:06:59.049+0000 7f6acfc5f640 -1 reset not still connected to 0x555c09ea0d00 2024-09-18T00:06:59.052 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:06:59.049+0000 7f6acfc5f640 -1 reset not still connected to 0x555c09ea1d40 2024-09-18T00:06:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-18T00:06:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:06:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:06:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:06:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:06:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:06:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:06:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:06:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:06:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:07:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:07:00.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3c 2024-09-18T00:07:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:07:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70352896.0,"last_snapshot_bytes":70352896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"} 2024-09-18T00:07:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:07:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:47 2024-09-18T00:07:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:07:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:07:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:07:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:07:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:06:57' 2024-09-18T00:07:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:07:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:07:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:07:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70352896.0,"last_snapshot_bytes":70352896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"} 2024-09-18T00:07:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:07:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:07:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:07:00.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:07:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:07:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:07:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:07:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:07:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:47' 2024-09-18T00:07:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-18 00:06:47' 2024-09-18T00:07:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:07:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:06:47' 2024-09-18T00:07:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:07:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:07:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:07:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d275e8cc-ba2c-4fd6-9649-c2f5e7ebfe3cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70352896.0,"last_snapshot_bytes":70352896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726617960,"remote_snapshot_timestamp":1726617960,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:06:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:06:57' 2024-09-18T00:07:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:07:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:07:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:06:47' 2024-09-18T00:07:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:07:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:07:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:07:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:07:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:07:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:07:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:07:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-18T00:07:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:07:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-18T00:07:00.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-18T00:07:01.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:07:02.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-18T00:07:02.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:07:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:07:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary10 2024-09-18T00:07:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:07:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary10 2024-09-18T00:07:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..10} 2024-09-18T00:07:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS=() 2024-09-18T00:07:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS=() 2024-09-18T00:07:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS=() 2024-09-18T00:07:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS=() 2024-09-18T00:07:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ RET=0 2024-09-18T00:07:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:07:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-18T00:07:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary1 2024-09-18T00:07:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-18T00:07:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:07:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-18T00:07:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:07:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:07:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:07:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-18T00:07:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:07:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:07:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:07:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-18T00:07:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:02.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-18T00:07:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:07:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-18T00:07:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary1': 2024-09-18T00:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 276e85855cd5 2024-09-18T00:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.276e85855cd5 2024-09-18T00:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:07:02 2024 2024-09-18T00:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:07:02 2024 2024-09-18T00:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:07:02 2024 2024-09-18T00:07:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: b89802c1-ca8a-4855-9327-8d4f21083a07 2024-09-18T00:07:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:07:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:07:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:07:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-18T00:07:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd0 2024-09-18T00:07:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:07:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-18T00:07:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:07:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:07:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:07:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 5c44b257-8d93-4bb1-bad3-66fe3735a94f 2024-09-18T00:07:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:07:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:07:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:07:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:07:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:07:03.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:07:03.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary1 2024-09-18T00:07:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-18T00:07:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:07:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary1 2024-09-18T00:07:03.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:07:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-18T00:07:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:07:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:07:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:07:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary1 2024-09-18T00:07:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-18T00:07:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary2 2024-09-18T00:07:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:07:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-18T00:07:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary1 2024-09-18T00:07:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-18T00:07:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:07:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-18T00:07:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:07:03.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:07:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:07:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-18T00:07:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:07:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:07:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:07:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-18T00:07:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-18T00:07:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:07:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-18T00:07:03.562 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:07:03.560+0000 7f6acfc5f640 -1 reset not still connected to 0x555c09ea0d00 2024-09-18T00:07:03.562 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:07:03.560+0000 7f6acfc5f640 -1 reset not still connected to 0x555c09ea1d40 2024-09-18T00:07:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:07:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1656 2024-09-18T00:07:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-18T00:07:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary2': 2024-09-18T00:07:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:07:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:07:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:07:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 277eccd07462 2024-09-18T00:07:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.277eccd07462 2024-09-18T00:07:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:07:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:07:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:07:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:07:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:07:03 2024 2024-09-18T00:07:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:07:03 2024 2024-09-18T00:07:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:07:03 2024 2024-09-18T00:07:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:07:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:07:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 218e33a2-8ff3-48a3-a4d1-2243247be64d 2024-09-18T00:07:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:07:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:07:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:07:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-18T00:07:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd1 2024-09-18T00:07:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:07:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-18T00:07:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:07:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:07:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:07:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: b46c8fa2-1601-4e8a-be04-f4b82ba62a00 2024-09-18T00:07:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:07:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:07:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:07:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:07:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:07:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:07:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary2 2024-09-18T00:07:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-18T00:07:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:07:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary2 2024-09-18T00:07:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:07:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-18T00:07:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:07:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:07:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:07:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:07:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary2 2024-09-18T00:07:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-18T00:07:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary3 2024-09-18T00:07:04.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-18T00:07:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary2 2024-09-18T00:07:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-18T00:07:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:07:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-18T00:07:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:07:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:07:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:07:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-18T00:07:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:07:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:07:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:07:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-18T00:07:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:04.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-18T00:07:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:07:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-18T00:07:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1659 2024-09-18T00:07:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:07:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-18T00:07:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary3': 2024-09-18T00:07:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:07:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:07:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:07:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 27899a737aae 2024-09-18T00:07:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.27899a737aae 2024-09-18T00:07:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:07:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:07:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:07:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:07:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:07:04 2024 2024-09-18T00:07:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:07:04 2024 2024-09-18T00:07:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:07:04 2024 2024-09-18T00:07:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:07:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:07:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 2cb77aa7-e9ed-4631-ae68-2c46fe019d9f 2024-09-18T00:07:04.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:07:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:07:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:07:04.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-18T00:07:04.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd2 2024-09-18T00:07:04.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:07:04.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-18T00:07:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:07:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:07:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:07:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: ead909d7-40f6-4b53-be3d-66eee8dc4a61 2024-09-18T00:07:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:07:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:07:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:07:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:07:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:07:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:07:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary3 2024-09-18T00:07:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-18T00:07:04.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:07:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary3 2024-09-18T00:07:04.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:07:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-18T00:07:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:07:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:07:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:07:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary3 2024-09-18T00:07:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:07:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-18T00:07:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary4 2024-09-18T00:07:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-18T00:07:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary3 2024-09-18T00:07:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-18T00:07:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:07:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-18T00:07:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:07:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:07:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:07:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-18T00:07:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:07:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:04.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:07:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:07:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-18T00:07:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:04.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-18T00:07:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:07:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-18T00:07:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:07:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1660 2024-09-18T00:07:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-18T00:07:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary4': 2024-09-18T00:07:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:07:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:07:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:07:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 279acf2ab02 2024-09-18T00:07:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.279acf2ab02 2024-09-18T00:07:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:07:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:07:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:07:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:07:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:07:04 2024 2024-09-18T00:07:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:07:04 2024 2024-09-18T00:07:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:07:04 2024 2024-09-18T00:07:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:07:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:07:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: b5b69d0c-e317-47f0-89bc-b0d989333a73 2024-09-18T00:07:05.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:07:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:07:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:07:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-18T00:07:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd3 2024-09-18T00:07:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:07:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-18T00:07:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:07:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:07:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:07:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 17625a64-045b-4923-9ecc-2f06cb9ea8ce 2024-09-18T00:07:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:07:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:07:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:07:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:07:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:07:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:07:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary4 2024-09-18T00:07:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-18T00:07:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:07:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary4 2024-09-18T00:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-18T00:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-18T00:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary5 2024-09-18T00:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary4 2024-09-18T00:07:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-18T00:07:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:07:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:07:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary4 2024-09-18T00:07:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-18T00:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-18T00:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:07:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:07:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:07:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-18T00:07:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:07:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:07:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:07:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-18T00:07:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:07:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-18T00:07:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:07:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-18T00:07:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:07:06.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1662 2024-09-18T00:07:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-18T00:07:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:07:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:07:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary5': 2024-09-18T00:07:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:07:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:07:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:07:06.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 27a94855aa8d 2024-09-18T00:07:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.27a94855aa8d 2024-09-18T00:07:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:07:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:07:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:07:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:07:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:07:06 2024 2024-09-18T00:07:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:07:06 2024 2024-09-18T00:07:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:07:06 2024 2024-09-18T00:07:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:07:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:07:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 630658f3-635d-4f74-99f3-bdef1657b63c 2024-09-18T00:07:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:07:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:07:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:07:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-18T00:07:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd4 2024-09-18T00:07:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:07:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-18T00:07:06.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:07:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:07:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:07:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: a6b6464b-0803-4d3e-8ae2-2a8ec6e29a6f 2024-09-18T00:07:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:07:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:07:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:07:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:07:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:07:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:07:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary5 2024-09-18T00:07:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-18T00:07:07.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:07:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary5 2024-09-18T00:07:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:07:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-18T00:07:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:07:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:07:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-18T00:07:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary6 2024-09-18T00:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary5 2024-09-18T00:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-18T00:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:07.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary5 2024-09-18T00:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-18T00:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-18T00:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:07:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:07:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:07:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:07:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-18T00:07:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:07:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:07:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:07:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-18T00:07:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:07.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:07:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:07.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-18T00:07:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:07:07.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-18T00:07:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:07:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:07:07.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:07:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1665 2024-09-18T00:07:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-18T00:07:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:07.831 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:07:07.829+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:07:07.831 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:07:07.829+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-18T00:07:08.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary6': 2024-09-18T00:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 27ba85b7865d 2024-09-18T00:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.27ba85b7865d 2024-09-18T00:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:07:07 2024 2024-09-18T00:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:07:07 2024 2024-09-18T00:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:07:07 2024 2024-09-18T00:07:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 92d2ac4c-65b0-4354-847f-ae86007ed627 2024-09-18T00:07:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:07:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:07:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:07:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-18T00:07:08.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd5 2024-09-18T00:07:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:07:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-18T00:07:08.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:07:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:07:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:07:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 3685115e-da85-4fd3-b15f-693ecfbbc6bb 2024-09-18T00:07:08.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:07:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:07:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:07:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:07:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:07:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:07:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary6 2024-09-18T00:07:08.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-18T00:07:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:07:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary6 2024-09-18T00:07:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:07:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-18T00:07:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:07:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:07:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:07:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary6 2024-09-18T00:07:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-18T00:07:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary7 2024-09-18T00:07:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:07:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-18T00:07:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:07:08.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:07:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary6 2024-09-18T00:07:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:07:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-18T00:07:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:07:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-18T00:07:08.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:07:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:07:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:07:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:07:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:08.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-18T00:07:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:07:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:07:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:07:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-18T00:07:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:07:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-18T00:07:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:07:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-18T00:07:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1664 2024-09-18T00:07:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:07:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:07:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:08.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1668 2024-09-18T00:07:08.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1667 2024-09-18T00:07:09.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:09.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:07:09.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-18T00:07:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary7': 2024-09-18T00:07:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:07:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:07:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:07:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 27cb6e4a98ba 2024-09-18T00:07:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.27cb6e4a98ba 2024-09-18T00:07:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:07:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:07:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:07:09.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:07:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:07:08 2024 2024-09-18T00:07:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:07:08 2024 2024-09-18T00:07:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:07:08 2024 2024-09-18T00:07:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:07:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:07:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 74b61702-1dd3-4fc5-a2c4-d9fdffc235df 2024-09-18T00:07:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:07:09.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:07:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:07:09.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-18T00:07:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:07:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:07:09.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd6 2024-09-18T00:07:09.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:07:09.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-18T00:07:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:07:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:07:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:07:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 25a19458-82ab-426a-839f-7c85c2c3cbb2 2024-09-18T00:07:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:07:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:07:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:07:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:07:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:07:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:07:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary7 2024-09-18T00:07:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-18T00:07:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:07:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary7 2024-09-18T00:07:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:07:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-18T00:07:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:07:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:07:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:07:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:07:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary7 2024-09-18T00:07:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-18T00:07:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary8 2024-09-18T00:07:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:07:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-18T00:07:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:07:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:07:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary7 2024-09-18T00:07:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:07:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-18T00:07:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-18T00:07:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:07:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:07:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:07:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:07:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-18T00:07:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:07:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:07:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:07:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-18T00:07:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:07:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-18T00:07:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:07:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-18T00:07:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1670 2024-09-18T00:07:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:07:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:07:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:07:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:07:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1672 2024-09-18T00:07:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-18T00:07:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary8': 2024-09-18T00:07:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:07:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:07:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:07:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 27df71f729fc 2024-09-18T00:07:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.27df71f729fc 2024-09-18T00:07:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:07:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:07:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:07:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:07:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:07:10 2024 2024-09-18T00:07:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:07:10 2024 2024-09-18T00:07:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:07:10 2024 2024-09-18T00:07:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:07:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:07:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 45ef8e7e-a26e-485b-868f-8d394feac98d 2024-09-18T00:07:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:07:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:07:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:07:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-18T00:07:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd7 2024-09-18T00:07:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:07:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-18T00:07:11.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:07:11.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:07:11.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:07:11.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 87d63d60-45ce-4831-a37e-10c176ff399f 2024-09-18T00:07:11.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:07:11.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:07:11.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:11.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:07:11.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:07:11.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:07:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:07:11.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary8 2024-09-18T00:07:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-18T00:07:11.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:07:11.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary8 2024-09-18T00:07:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:07:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-18T00:07:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:07:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:07:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:07:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:07:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary8 2024-09-18T00:07:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-18T00:07:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:11.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary9 2024-09-18T00:07:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:07:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:07:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-18T00:07:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary8 2024-09-18T00:07:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-18T00:07:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:07:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:07:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-18T00:07:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:07:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:07:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:07:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:07:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-18T00:07:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:07:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:07:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:07:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-18T00:07:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:07:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-18T00:07:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:07:11.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-18T00:07:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1671 2024-09-18T00:07:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:07:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:07:11.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:07:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:07:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:07:11.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:07:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-18T00:07:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1675 2024-09-18T00:07:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary9': 2024-09-18T00:07:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:07:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:07:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:07:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 27f17f376c42 2024-09-18T00:07:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.27f17f376c42 2024-09-18T00:07:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:07:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:07:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:07:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:07:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:07:11 2024 2024-09-18T00:07:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:07:11 2024 2024-09-18T00:07:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:07:11 2024 2024-09-18T00:07:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:07:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:07:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: b7f435e7-f034-4987-af7c-ed84c4d984bf 2024-09-18T00:07:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:07:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:07:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:07:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-18T00:07:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd8 2024-09-18T00:07:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:07:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-18T00:07:12.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:07:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:07:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:07:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 3ba2411f-60c1-4c53-ab86-4b7f9eeb662c 2024-09-18T00:07:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:07:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:07:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:07:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:07:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:07:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:07:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:12.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary9 2024-09-18T00:07:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-18T00:07:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:07:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary9 2024-09-18T00:07:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:07:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-18T00:07:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:07:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:07:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:07:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:07:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary9 2024-09-18T00:07:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-18T00:07:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:07:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary10 2024-09-18T00:07:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:07:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-18T00:07:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary9 2024-09-18T00:07:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-18T00:07:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:07:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-18T00:07:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:07:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:07:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:07:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:07:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-18T00:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-18T00:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-18T00:07:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:07:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-18T00:07:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:07:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:07:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1674 2024-09-18T00:07:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:07:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-18T00:07:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1679 2024-09-18T00:07:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary10': 2024-09-18T00:07:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:07:12.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:07:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:07:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 28053c66af56 2024-09-18T00:07:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.28053c66af56 2024-09-18T00:07:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:07:12.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:07:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:07:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:07:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:07:12 2024 2024-09-18T00:07:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:07:12 2024 2024-09-18T00:07:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:07:12 2024 2024-09-18T00:07:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:07:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:07:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 7d207094-b0f0-4686-a9cf-144f851a07f8 2024-09-18T00:07:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:07:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:07:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:07:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-18T00:07:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd9 2024-09-18T00:07:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:07:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-18T00:07:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:07:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:07:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:07:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 962fe22f-6347-43ad-9ab5-04b8becac6ec 2024-09-18T00:07:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:07:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:07:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:07:13.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:07:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:07:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:07:13.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:07:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary10 2024-09-18T00:07:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-18T00:07:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:07:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary10 2024-09-18T00:07:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:07:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-18T00:07:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:07:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:13.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:07:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:07:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary10 2024-09-18T00:07:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 94973 2024-09-18T00:07:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:07:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary10 2024-09-18T00:07:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-18T00:07:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:07:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-18T00:07:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1681 2024-09-18T00:07:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1677 2024-09-18T00:07:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1678 2024-09-18T00:07:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:07:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:07:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:13.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:13.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1682 2024-09-18T00:07:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:07:13.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:07:13.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:07:14.698 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:07:14.697+0000 7f9bfab49640 -1 reset not still connected to 0x55cd38060c30 2024-09-18T00:07:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:07:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:07:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:07:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:07:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:07:15.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1684 2024-09-18T00:07:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:15.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:07:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:07:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:07:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:07:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:07:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:07:16.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1683 2024-09-18T00:07:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1686 2024-09-18T00:07:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1685 2024-09-18T00:07:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:16.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:07:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:16.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:07:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:07:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:07:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:07:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:07:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:07:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:07:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:07:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1687 2024-09-18T00:07:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1688 2024-09-18T00:07:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:17.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:18.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1692 2024-09-18T00:07:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:07:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:07:18.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:07:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1690 2024-09-18T00:07:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1691 2024-09-18T00:07:18.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:18.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:19.392 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:07:19.390+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:07:19.392 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:07:19.390+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-18T00:07:19.393 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:07:19.390+0000 7f706e974640 -1 reset not still connected to 0x55df2f33ef70 2024-09-18T00:07:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:07:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:07:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:07:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:07:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:07:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:07:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:07:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1689 2024-09-18T00:07:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:20.108 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:07:20.106+0000 7f9bfab49640 -1 reset not still connected to 0x55cd32063930 2024-09-18T00:07:20.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:07:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:07:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:07:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:07:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:07:20.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:07:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1693 2024-09-18T00:07:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:20.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1696 2024-09-18T00:07:20.770 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:07:20.768+0000 7f706e974640 -1 reset not still connected to 0x55df20339d40 2024-09-18T00:07:20.770 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:07:20.768+0000 7f706e974640 -1 reset not still connected to 0x55df267d1ba0 2024-09-18T00:07:20.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1695 2024-09-18T00:07:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1694 2024-09-18T00:07:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:21.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:07:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:07:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:07:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:07:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:07:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:07:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:07:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1697 2024-09-18T00:07:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1698 2024-09-18T00:07:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:07:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:07:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1701 2024-09-18T00:07:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1700 2024-09-18T00:07:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1699 2024-09-18T00:07:23.676 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:07:23.673+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0687dd40 2024-09-18T00:07:23.676 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:07:23.673+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0986e9c0 2024-09-18T00:07:23.676 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:07:23.673+0000 7f9bfab49640 -1 reset not still connected to 0x55cd2563f110 2024-09-18T00:07:23.676 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:07:23.673+0000 7f9bfab49640 -1 reset not still connected to 0x55cd2c54d5f0 2024-09-18T00:07:23.676 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:07:23.673+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3147c000 2024-09-18T00:07:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:07:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:23.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:07:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:07:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:23.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1702 2024-09-18T00:07:23.709 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:07:23.707+0000 7fd413089640 -1 reset not still connected to 0x5604a9ed3380 2024-09-18T00:07:23.709 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:07:23.707+0000 7fd413089640 -1 reset not still connected to 0x5604ab615040 2024-09-18T00:07:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:07:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:23.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:07:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:07:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:07:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:07:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:07:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:07:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1703 2024-09-18T00:07:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:07:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:07:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:07:25.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1704 2024-09-18T00:07:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:07:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:07:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:07:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:25.930 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:07:25.927+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0ef59040 2024-09-18T00:07:25.930 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:07:25.927+0000 7f9bfab49640 -1 reset not still connected to 0x55cd38204ea0 2024-09-18T00:07:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1706 2024-09-18T00:07:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:07:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:07:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:07:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1705 2024-09-18T00:07:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:07:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:07:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:07:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:07:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:07:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:07:26.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1707 2024-09-18T00:07:27.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:28.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1712 2024-09-18T00:07:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1708 2024-09-18T00:07:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1710 2024-09-18T00:07:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1709 2024-09-18T00:07:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1711 2024-09-18T00:07:28.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:07:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:07:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:07:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:07:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:07:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:07:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:29.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:07:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:07:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:07:30.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1713 2024-09-18T00:07:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:07:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:07:30.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:07:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1716 2024-09-18T00:07:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1714 2024-09-18T00:07:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1715 2024-09-18T00:07:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:07:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:07:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:07:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:07:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:07:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:07:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:31.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:07:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:07:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:07:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:07:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:07:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:07:32.376 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:07:32.375+0000 7f0114255640 -1 reset not still connected to 0x557d0a065e10 2024-09-18T00:07:32.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1717 2024-09-18T00:07:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:32.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1721 2024-09-18T00:07:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:07:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:07:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:07:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1718 2024-09-18T00:07:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1720 2024-09-18T00:07:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1722 2024-09-18T00:07:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1719 2024-09-18T00:07:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:07:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:07:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:07:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:07:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:07:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:07:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:34.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:07:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1723 2024-09-18T00:07:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1725 2024-09-18T00:07:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:07:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:07:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:07:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:07:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:07:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:07:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1726 2024-09-18T00:07:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1724 2024-09-18T00:07:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:07:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:07:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:07:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:36.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:07:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:07:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:07:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:07:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:07:36.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:07:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1728 2024-09-18T00:07:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1727 2024-09-18T00:07:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:07:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:07:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:07:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:07:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:38.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:07:38.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1732 2024-09-18T00:07:38.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1729 2024-09-18T00:07:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1731 2024-09-18T00:07:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1730 2024-09-18T00:07:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:07:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:07:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:07:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:07:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1733 2024-09-18T00:07:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1734 2024-09-18T00:07:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:07:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:07:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:07:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:07:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:07:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:07:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1735 2024-09-18T00:07:41.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1736 2024-09-18T00:07:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:41.622 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:07:41.620+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3e1a7110 2024-09-18T00:07:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:07:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:07:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:07:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:07:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:07:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:07:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:07:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1738 2024-09-18T00:07:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1737 2024-09-18T00:07:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:07:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:07:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1741 2024-09-18T00:07:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1740 2024-09-18T00:07:43.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:07:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:07:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:07:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:43.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:07:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:07:43.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1742 2024-09-18T00:07:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1739 2024-09-18T00:07:44.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:44.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:07:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:44.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:07:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:07:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:07:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:07:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1743 2024-09-18T00:07:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:07:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:07:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:07:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:07:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:07:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:07:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1746 2024-09-18T00:07:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1744 2024-09-18T00:07:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1745 2024-09-18T00:07:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:07:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:07:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:07:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:07:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:07:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:07:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:07:47.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1747 2024-09-18T00:07:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1748 2024-09-18T00:07:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:07:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:07:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1751 2024-09-18T00:07:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1750 2024-09-18T00:07:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:07:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:07:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:07:48.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1752 2024-09-18T00:07:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:49.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1749 2024-09-18T00:07:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:07:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:49.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:07:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:07:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:07:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:07:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:07:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:07:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1753 2024-09-18T00:07:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:07:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:07:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:07:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:07:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:07:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:07:50.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1755 2024-09-18T00:07:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1756 2024-09-18T00:07:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1754 2024-09-18T00:07:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:07:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:07:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:07:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:07:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:51.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:07:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:51.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:07:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:51.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:07:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1757 2024-09-18T00:07:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:07:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:07:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1758 2024-09-18T00:07:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:52.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1759 2024-09-18T00:07:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1760 2024-09-18T00:07:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1761 2024-09-18T00:07:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:07:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:07:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:07:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:53.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1762 2024-09-18T00:07:53.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:07:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:07:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:07:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:54.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:07:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:07:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:07:54.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:07:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1763 2024-09-18T00:07:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:07:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:55.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:55.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:07:55.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:07:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:07:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:07:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:07:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1765 2024-09-18T00:07:56.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1766 2024-09-18T00:07:56.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1764 2024-09-18T00:07:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:07:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:07:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:07:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:07:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:07:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:07:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:56.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:07:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:07:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:07:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:07:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:07:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:07:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1767 2024-09-18T00:07:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1768 2024-09-18T00:07:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1770 2024-09-18T00:07:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1772 2024-09-18T00:07:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1769 2024-09-18T00:07:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1771 2024-09-18T00:07:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:07:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:07:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:07:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:58.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:07:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:07:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:07:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:07:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:07:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:07:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:07:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:07:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:07:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:07:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:07:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:07:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:08:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1773 2024-09-18T00:08:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:08:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:00.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:00.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:08:00.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:08:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1776 2024-09-18T00:08:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:08:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:08:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:08:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1775 2024-09-18T00:08:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:08:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:08:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:08:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1774 2024-09-18T00:08:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:08:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:08:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:08:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:08:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:08:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:08:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:08:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:08:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:08:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1777 2024-09-18T00:08:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1778 2024-09-18T00:08:02.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1782 2024-09-18T00:08:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:08:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:08:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:08:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1780 2024-09-18T00:08:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1781 2024-09-18T00:08:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:03.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:08:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:08:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:08:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1779 2024-09-18T00:08:04.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:08:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:08:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:08:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:08:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:08:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:08:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1784 2024-09-18T00:08:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1783 2024-09-18T00:08:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:08:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:08:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:08:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:08:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:08:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:08:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:06.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1785 2024-09-18T00:08:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1786 2024-09-18T00:08:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:08:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:08:06.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:08:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:08:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:08:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:08:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:08:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:08:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:08:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:08:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:08:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:08:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1788 2024-09-18T00:08:07.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1787 2024-09-18T00:08:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:08.683 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:08:08.681+0000 7f9bfab49640 -1 reset not still connected to 0x55cd2563f110 2024-09-18T00:08:08.683 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:08:08.681+0000 7f9bfab49640 -1 reset not still connected to 0x55cd2c54d5f0 2024-09-18T00:08:08.683 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:08:08.681+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3147c000 2024-09-18T00:08:08.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1791 2024-09-18T00:08:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:08:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:08:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:08:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:08:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:08:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:08:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:08.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1792 2024-09-18T00:08:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:09.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1789 2024-09-18T00:08:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1790 2024-09-18T00:08:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:08:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:08:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:08:09.701 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:08:09.700+0000 7f9bfab49640 -1 reset not still connected to 0x55cd2c54d5f0 2024-09-18T00:08:09.702 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:08:09.700+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3147c000 2024-09-18T00:08:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:08:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:09.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:09.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:08:09.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:08:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1794 2024-09-18T00:08:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1793 2024-09-18T00:08:10.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:10.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:08:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:08:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:08:10.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:10.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:08:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:08:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:08:11.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1795 2024-09-18T00:08:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1796 2024-09-18T00:08:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:11.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:08:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:08:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:08:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:08:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:08:11.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:08:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:08:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:08:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:08:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:08:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:08:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:08:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1797 2024-09-18T00:08:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1798 2024-09-18T00:08:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:13.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1800 2024-09-18T00:08:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:08:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:08:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:08:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:08:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:08:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:08:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1801 2024-09-18T00:08:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1802 2024-09-18T00:08:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1799 2024-09-18T00:08:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:08:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:08:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:08:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:08:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:08:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:08:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1804 2024-09-18T00:08:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1803 2024-09-18T00:08:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:15.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:08:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:08:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:08:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:08:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:08:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:08:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1805 2024-09-18T00:08:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1806 2024-09-18T00:08:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:08:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:08:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:08:16.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:16.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:08:16.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:16.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:08:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:08:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:08:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:08:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:08:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1807 2024-09-18T00:08:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1808 2024-09-18T00:08:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:08:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:08:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:08:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:18.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1809 2024-09-18T00:08:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1810 2024-09-18T00:08:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1811 2024-09-18T00:08:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:08:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:08:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:08:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:08:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:08:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:08:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:18.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1812 2024-09-18T00:08:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:08:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:08:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:08:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:08:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:08:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:08:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1814 2024-09-18T00:08:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1813 2024-09-18T00:08:20.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:08:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:08:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:08:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:08:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:08:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:08:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:20.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1816 2024-09-18T00:08:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1815 2024-09-18T00:08:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:08:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:08:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:08:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:08:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:08:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:08:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:08:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:08:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:08:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:08:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:08:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:08:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1817 2024-09-18T00:08:22.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1818 2024-09-18T00:08:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1821 2024-09-18T00:08:23.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1820 2024-09-18T00:08:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:23.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:08:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:08:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:08:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1822 2024-09-18T00:08:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:08:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:08:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:08:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1819 2024-09-18T00:08:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:23.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:08:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:08:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:08:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:08:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:08:24.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:08:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1824 2024-09-18T00:08:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1823 2024-09-18T00:08:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:08:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:08:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:08:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:08:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:08:25.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:08:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1826 2024-09-18T00:08:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1825 2024-09-18T00:08:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:08:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:08:26.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:08:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:08:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:26.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:08:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:08:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:08:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:26.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:08:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:08:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:08:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:08:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:08:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1828 2024-09-18T00:08:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1827 2024-09-18T00:08:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1832 2024-09-18T00:08:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1830 2024-09-18T00:08:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1831 2024-09-18T00:08:28.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1829 2024-09-18T00:08:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:08:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:08:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:08:28.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:08:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:08:28.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:08:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:08:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:08:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:08:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:08:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:08:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:08:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:08:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:08:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:08:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:08:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:08:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:08:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1833 2024-09-18T00:08:30.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1834 2024-09-18T00:08:30.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:31.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1836 2024-09-18T00:08:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1835 2024-09-18T00:08:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:08:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:08:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:08:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:08:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:08:31.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:08:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:08:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:08:31.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:08:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:08:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:08:31.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:08:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1837 2024-09-18T00:08:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1839 2024-09-18T00:08:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1842 2024-09-18T00:08:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1840 2024-09-18T00:08:33.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1838 2024-09-18T00:08:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1841 2024-09-18T00:08:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:33.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:08:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:08:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:08:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:08:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:08:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:08:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:08:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:08:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:08:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:08:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:08:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:08:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1844 2024-09-18T00:08:35.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1843 2024-09-18T00:08:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:35.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:08:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:08:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:08:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1845 2024-09-18T00:08:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:08:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:08:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:08:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:08:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:08:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:08:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:08:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:08:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:08:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:08:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:08:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:08:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:08:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:08:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:08:36.806 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:08:36.804+0000 7f0114255640 -1 reset not still connected to 0x557d0c7bf2b0 2024-09-18T00:08:36.806 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:08:36.804+0000 7f0114255640 -1 reset not still connected to 0x557d10dd88f0 2024-09-18T00:08:37.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1846 2024-09-18T00:08:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1847 2024-09-18T00:08:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1849 2024-09-18T00:08:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:08:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:08:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:08:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:08:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:08:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:08:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1851 2024-09-18T00:08:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1852 2024-09-18T00:08:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1848 2024-09-18T00:08:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1850 2024-09-18T00:08:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:08:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:08:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:08:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1854 2024-09-18T00:08:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1853 2024-09-18T00:08:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:40.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:08:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:08:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:08:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:08:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:40.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:08:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:08:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:08:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:08:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:08:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1855 2024-09-18T00:08:41.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1856 2024-09-18T00:08:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:08:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:08:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:08:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:08:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:08:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:08:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:08:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:08:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:08:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:08:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:08:41.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:08:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1857 2024-09-18T00:08:42.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1858 2024-09-18T00:08:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:42.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:43.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1860 2024-09-18T00:08:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1859 2024-09-18T00:08:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1862 2024-09-18T00:08:43.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1861 2024-09-18T00:08:43.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:43.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:08:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:08:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:08:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:08:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:08:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:08:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:08:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:08:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:08:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:08:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:08:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:08:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1864 2024-09-18T00:08:45.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1863 2024-09-18T00:08:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:08:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:08:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:08:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:08:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:08:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:08:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1866 2024-09-18T00:08:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1865 2024-09-18T00:08:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:08:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:08:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:08:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:08:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:08:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:08:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:08:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:08:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:08:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:08:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:08:46.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:08:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1867 2024-09-18T00:08:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1868 2024-09-18T00:08:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1872 2024-09-18T00:08:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1871 2024-09-18T00:08:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1870 2024-09-18T00:08:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1869 2024-09-18T00:08:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:08:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:08:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:08:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:48.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:08:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:08:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:08:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:08:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:08:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:08:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:08:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:08:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:08:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1874 2024-09-18T00:08:50.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1873 2024-09-18T00:08:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:08:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:08:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:08:50.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:50.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:08:50.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:50.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:50.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:08:50.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:08:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:50.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:51.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1875 2024-09-18T00:08:51.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1876 2024-09-18T00:08:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:08:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:08:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:08:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:08:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:08:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:08:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:08:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:08:51.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:08:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:08:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:51.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:08:51.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:08:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1877 2024-09-18T00:08:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1878 2024-09-18T00:08:52.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1880 2024-09-18T00:08:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1882 2024-09-18T00:08:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1879 2024-09-18T00:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:08:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:08:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:08:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:08:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:08:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:08:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1881 2024-09-18T00:08:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:08:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:54.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:08:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:08:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:08:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:08:54.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:08:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1883 2024-09-18T00:08:55.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1884 2024-09-18T00:08:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:08:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:08:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:08:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:08:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:08:55.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:08:55.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:56.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1885 2024-09-18T00:08:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1886 2024-09-18T00:08:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:08:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:08:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:08:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:08:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:08:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:08:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:08:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:08:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:08:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:08:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:08:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:08:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1887 2024-09-18T00:08:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1888 2024-09-18T00:08:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1889 2024-09-18T00:08:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1891 2024-09-18T00:08:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:08:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:08:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:08:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:08:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:08:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:08:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:58.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1892 2024-09-18T00:08:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1890 2024-09-18T00:08:59.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:08:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:08:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:08:59.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:08:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:08:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:08:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:08:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:08:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:08:59.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:09:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1893 2024-09-18T00:09:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1894 2024-09-18T00:09:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:09:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:09:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:09:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1896 2024-09-18T00:09:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1895 2024-09-18T00:09:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:09:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:09:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:09:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:09:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:01.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:09:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:09:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:09:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:09:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:09:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:01.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:09:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:09:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:09:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:09:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:09:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:09:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1899 2024-09-18T00:09:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1900 2024-09-18T00:09:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1901 2024-09-18T00:09:03.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1898 2024-09-18T00:09:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1897 2024-09-18T00:09:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:09:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:09:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:09:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:09:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:09:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:09:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:03.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:03.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1902 2024-09-18T00:09:03.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:09:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:09:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:09:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:09:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:09:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:09:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1904 2024-09-18T00:09:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1903 2024-09-18T00:09:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:09:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:09:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:09:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1905 2024-09-18T00:09:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1906 2024-09-18T00:09:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:09:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:09:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:09:06.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:09:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:09:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:09:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:09:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:09:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:09:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:09:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:09:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:09:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:09:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:09:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:09:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1907 2024-09-18T00:09:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1911 2024-09-18T00:09:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:09:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:09:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:09:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:09:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:09:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:09:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1908 2024-09-18T00:09:08.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1910 2024-09-18T00:09:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1909 2024-09-18T00:09:08.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1912 2024-09-18T00:09:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:08.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:09:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:09:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:09:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:09:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:09:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:09:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1913 2024-09-18T00:09:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1914 2024-09-18T00:09:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:09:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:09:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:09:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:09:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:09:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:09:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1915 2024-09-18T00:09:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1916 2024-09-18T00:09:11.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:09:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:09:11.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:09:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:09:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:09:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:09:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:09:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:09:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:09:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:09:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:09:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:09:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1917 2024-09-18T00:09:12.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1918 2024-09-18T00:09:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:12.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:09:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:09:13.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:09:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:09:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:09:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:09:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1920 2024-09-18T00:09:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1921 2024-09-18T00:09:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1922 2024-09-18T00:09:13.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1919 2024-09-18T00:09:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:09:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:09:14.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:09:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:09:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:09:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:09:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1924 2024-09-18T00:09:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1923 2024-09-18T00:09:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:09:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:09:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:09:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:09:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:09:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:09:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1926 2024-09-18T00:09:16.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1925 2024-09-18T00:09:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:09:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:09:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:09:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:09:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:09:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:09:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:09:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:09:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:09:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:09:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:09:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:09:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1928 2024-09-18T00:09:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1927 2024-09-18T00:09:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:09:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:09:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:09:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:09:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:09:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:09:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1931 2024-09-18T00:09:18.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1932 2024-09-18T00:09:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1930 2024-09-18T00:09:18.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:18.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1929 2024-09-18T00:09:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:09:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:09:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:09:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:09:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:09:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:09:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1934 2024-09-18T00:09:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1933 2024-09-18T00:09:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:20.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:09:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:09:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:09:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:09:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:09:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:09:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1936 2024-09-18T00:09:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1935 2024-09-18T00:09:21.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:09:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:09:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:09:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:09:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:09:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:09:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:09:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:09:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:09:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1937 2024-09-18T00:09:22.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1938 2024-09-18T00:09:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:22.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:09:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:09:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:09:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1939 2024-09-18T00:09:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1940 2024-09-18T00:09:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1941 2024-09-18T00:09:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:09:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:09:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:09:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:09:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:09:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:09:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1942 2024-09-18T00:09:23.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:09:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:24.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:09:24.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:09:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:09:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:09:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:09:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1943 2024-09-18T00:09:25.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1944 2024-09-18T00:09:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:09:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:09:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:09:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:09:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:09:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:09:26.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1945 2024-09-18T00:09:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1946 2024-09-18T00:09:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:09:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:09:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:09:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:09:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:09:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:09:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:09:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:09:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:09:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:09:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:09:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:09:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1948 2024-09-18T00:09:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1947 2024-09-18T00:09:27.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1951 2024-09-18T00:09:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1952 2024-09-18T00:09:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1950 2024-09-18T00:09:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1949 2024-09-18T00:09:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:09:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:09:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:09:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:09:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:09:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:09:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:09:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:09:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:09:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:09:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:09:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:09:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1954 2024-09-18T00:09:30.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1953 2024-09-18T00:09:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:09:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:09:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:09:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:09:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:09:30.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:09:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1956 2024-09-18T00:09:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1955 2024-09-18T00:09:31.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:09:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 95105 2024-09-18T00:09:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:09:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:09:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:09:31.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:31.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:09:31.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:31.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:31.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:09:31.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:09:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:09:31.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 95245 2024-09-18T00:09:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1958 2024-09-18T00:09:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1957 2024-09-18T00:09:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:32.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1959 2024-09-18T00:09:33.644 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1960 2024-09-18T00:09:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:09:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:09:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:09:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:09:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 95380 2024-09-18T00:09:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:09:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:09:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:09:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:09:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 95532 2024-09-18T00:09:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1961 2024-09-18T00:09:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:09:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 95684 2024-09-18T00:09:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:09:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:09:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:09:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1962 2024-09-18T00:09:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:09:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:09:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:09:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:09:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:09:36.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:09:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:09:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 95858 2024-09-18T00:09:36.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1963 2024-09-18T00:09:37.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1964 2024-09-18T00:09:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:09:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 96011 2024-09-18T00:09:38.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:09:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:09:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 96184 2024-09-18T00:09:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:09:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 96337 2024-09-18T00:09:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:09:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:09:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 94974 2024-09-18T00:12:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:12:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:12:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:12:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 95106 2024-09-18T00:12:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:12:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:12:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:12:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 95246 2024-09-18T00:12:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:12:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:12:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:12:05.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 95381 2024-09-18T00:12:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:12:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:12:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:12:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 95533 2024-09-18T00:12:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:12:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:12:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:12:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 95685 2024-09-18T00:12:07.393 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:12:07.390+0000 7f0114255640 -1 reset not still connected to 0x557d0a065e10 2024-09-18T00:12:07.393 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:12:07.390+0000 7f0114255640 -1 reset not still connected to 0x557d0b465110 2024-09-18T00:12:07.396 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:12:07.393+0000 7f0114255640 -1 reset not still connected to 0x557d0b465860 2024-09-18T00:12:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:12:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:12:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:12:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 95859 2024-09-18T00:12:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:12:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:12:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:12:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 96012 2024-09-18T00:12:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:12:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:12:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:12:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 96185 2024-09-18T00:12:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:12:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:12:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:12:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 96338 2024-09-18T00:12:13.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:12:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:12:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:12:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:12:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:12:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:12:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 1 2024-09-18T00:12:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:12:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd0 2024-09-18T00:12:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:12:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary1 2024-09-18T00:12:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary1 2024-09-18T00:12:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:12:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 2 2024-09-18T00:12:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary1 2024-09-18T00:12:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:12:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd1 2024-09-18T00:12:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary2 2024-09-18T00:12:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary2 2024-09-18T00:12:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:12:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary2 2024-09-18T00:12:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:12:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 3 2024-09-18T00:12:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd2 2024-09-18T00:12:13.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary3 2024-09-18T00:12:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary3 2024-09-18T00:12:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:12:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:12:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary3 2024-09-18T00:12:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:12:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 4 2024-09-18T00:12:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd3 2024-09-18T00:12:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary4 2024-09-18T00:12:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:12:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary4 2024-09-18T00:12:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:12:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary4 2024-09-18T00:12:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:12:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 5 2024-09-18T00:12:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd4 2024-09-18T00:12:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:12:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary5 2024-09-18T00:12:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary5 2024-09-18T00:12:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:12:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary5 2024-09-18T00:12:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:12:13.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 6 2024-09-18T00:12:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd5 2024-09-18T00:12:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:12:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary6 2024-09-18T00:12:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary6 2024-09-18T00:12:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:12:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary6 2024-09-18T00:12:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 7 2024-09-18T00:12:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:12:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd6 2024-09-18T00:12:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary7 2024-09-18T00:12:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:12:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary7 2024-09-18T00:12:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:12:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary7 2024-09-18T00:12:13.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:12:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:12:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:12:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:12:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:12:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 103109 2024-09-18T00:12:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 10 2024-09-18T00:12:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd9 2024-09-18T00:12:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary10 2024-09-18T00:12:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary10 2024-09-18T00:12:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:12:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary10 2024-09-18T00:12:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 8 2024-09-18T00:12:13.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd7 2024-09-18T00:12:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 9 2024-09-18T00:12:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary8 2024-09-18T00:12:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd8 2024-09-18T00:12:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary8 2024-09-18T00:12:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary9 2024-09-18T00:12:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary9 2024-09-18T00:12:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:12:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary9 2024-09-18T00:12:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:12:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary8 2024-09-18T00:12:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-18T00:12:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-18T00:12:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-18T00:12:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-18T00:12:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-18T00:12:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-18T00:12:13.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-18T00:12:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-18T00:12:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-18T00:12:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-18T00:12:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=7696752a39825087a61f37dc7bec9dbd 2024-09-18T00:12:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-18T00:12:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=c557fd6d3e614c6f46130e7fe7fa64e5 2024-09-18T00:12:43.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-18T00:12:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-18T00:12:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:12:43.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-18T00:12:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-18T00:12:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:12:43.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-18T00:12:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=3026604cb5901059dbc3227982b962c9 2024-09-18T00:12:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-18T00:12:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=1811bf9394c11175b9dfdbdfbd03eb6a 2024-09-18T00:12:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-18T00:12:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=3d328b5784c5973b7bf9bd82b90d2460 2024-09-18T00:12:44.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-18T00:12:44.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-18T00:12:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:12:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-18T00:12:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-18T00:12:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:12:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-18T00:12:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-18T00:12:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:12:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-18T00:12:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=fa0fbb6df9f356af18c880b8b062885f 2024-09-18T00:12:44.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-18T00:12:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-18T00:12:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:12:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-18T00:12:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=74f6b725c035779624a9523491a58281 2024-09-18T00:12:44.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-18T00:12:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=450d6a56e3992d09f637b15c9220ac0a 2024-09-18T00:12:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-18T00:12:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-18T00:12:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:12:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-18T00:12:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-18T00:12:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:12:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-18T00:12:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=1b2a5ec7adddad4dcddbaa2057baa597 2024-09-18T00:12:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-18T00:12:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=348bf5ba5c4e2ff1a61446075e5eb795 2024-09-18T00:12:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-18T00:12:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-18T00:12:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:12:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-18T00:12:44.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-18T00:12:44.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:44.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:44.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:12:44.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-18T00:12:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:12:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:12:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:12:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:12:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-18T00:12:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:12:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:12:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-18T00:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:45.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-18T00:12:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:12:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-18T00:12:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:12:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-18T00:12:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:12:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:46.259 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:12:46.257+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4270 2024-09-18T00:12:46.259 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:12:46.257+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4340 2024-09-18T00:12:46.259 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:12:46.257+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb52b0 2024-09-18T00:12:46.261 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:12:46.259+0000 7f706e974640 -1 reset not still connected to 0x55df2f247930 2024-09-18T00:12:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:12:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:12:46.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:12:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-18T00:12:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:12:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:12:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-18T00:12:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:12:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:12:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-18T00:12:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:12:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-18T00:12:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:12:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-18T00:12:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:12:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-18T00:12:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:12:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:12:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:12:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-18T00:12:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:12:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-18T00:12:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:46.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:12:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:12:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-18T00:12:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:12:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:12:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-18T00:12:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:12:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:46.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:46.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:12:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:12:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 92d2ac4c-65b0-4354-847f-ae86007ed627 2024-09-18T00:12:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:12:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:12:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:42 2024-09-18T00:12:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:12:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:12:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:46' 2024-09-18T00:12:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:12:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:12:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:12:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 218e33a2-8ff3-48a3-a4d1-2243247be64d 2024-09-18T00:12:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:12:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:12:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:42 2024-09-18T00:12:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:12:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:12:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:46' 2024-09-18T00:12:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:12:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:12:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:12:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 2cb77aa7-e9ed-4631-ae68-2c46fe019d9f 2024-09-18T00:12:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:12:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:12:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:42 2024-09-18T00:12:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:12:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:12:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:46' 2024-09-18T00:12:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:12:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:12:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:12:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 45ef8e7e-a26e-485b-868f-8d394feac98d 2024-09-18T00:12:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:12:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:12:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:42 2024-09-18T00:12:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:12:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:12:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:46' 2024-09-18T00:12:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:12:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:12:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:12:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 74b61702-1dd3-4fc5-a2c4-d9fdffc235df 2024-09-18T00:12:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:12:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:12:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:42 2024-09-18T00:12:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:12:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:12:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:46' 2024-09-18T00:12:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:12:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:12:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-18T00:12:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:12:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-18T00:12:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-18T00:12:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:12:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-18T00:12:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-18T00:12:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:12:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:47.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:12:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-18T00:12:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:12:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:12:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:12:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: b5b69d0c-e317-47f0-89bc-b0d989333a73 2024-09-18T00:12:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:12:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:12:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-18T00:12:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:12:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:12:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:12:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:12:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:12:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 630658f3-635d-4f74-99f3-bdef1657b63c 2024-09-18T00:12:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:12:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:12:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:47.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:12:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:47.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:47.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:47.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:47.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:12:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:47.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:47.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:47.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:47.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:12:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: b89802c1-ca8a-4855-9327-8d4f21083a07 2024-09-18T00:12:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:12:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:12:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:12:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:12:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:12:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 7d207094-b0f0-4686-a9cf-144f851a07f8 2024-09-18T00:12:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:12:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:12:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:12:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:12:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: b7f435e7-f034-4987-af7c-ed84c4d984bf 2024-09-18T00:12:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:12:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:12:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:47.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:12:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:47.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:12:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:47.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:47.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:12:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:47.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47' 2024-09-18T00:12:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:12:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:12:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-18T00:12:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:12:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47' 2024-09-18T00:12:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:12:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:12:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-18T00:12:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:12:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47' 2024-09-18T00:12:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:12:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:47.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:12:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:47.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-18T00:12:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:12:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47' 2024-09-18T00:12:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:12:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:47.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:12:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-18T00:12:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:12:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47' 2024-09-18T00:12:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:12:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:12:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:47.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-18T00:12:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:12:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:47.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-18T00:12:48.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:12:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:48.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-18T00:12:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-18T00:12:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:12:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:48.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:48.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-18T00:12:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:12:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: b5b69d0c-e317-47f0-89bc-b0d989333a73 2024-09-18T00:12:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:12:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:12:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:12:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:48.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:48.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:48.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:48.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:12:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-18T00:12:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:12:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-18T00:12:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-18T00:12:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:12:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:12:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-18T00:12:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:12:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-18T00:12:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-18T00:12:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:12:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-18T00:12:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-18T00:12:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:12:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:12:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:12:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 630658f3-635d-4f74-99f3-bdef1657b63c 2024-09-18T00:12:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:12:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:12:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:12:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:12:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:49.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:12:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: b89802c1-ca8a-4855-9327-8d4f21083a07 2024-09-18T00:12:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:12:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:12:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:49.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:12:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 92d2ac4c-65b0-4354-847f-ae86007ed627 2024-09-18T00:12:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:12:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:12:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:12:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:12:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:12:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-18T00:12:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:12:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:12:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:12:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 7d207094-b0f0-4686-a9cf-144f851a07f8 2024-09-18T00:12:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:12:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:12:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:49.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:12:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:12:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: b7f435e7-f034-4987-af7c-ed84c4d984bf 2024-09-18T00:12:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:12:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:12:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:49.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:12:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:12:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:12:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47' 2024-09-18T00:12:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:12:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-18T00:12:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:12:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:12:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:12:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 218e33a2-8ff3-48a3-a4d1-2243247be64d 2024-09-18T00:12:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:12:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:12:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:12:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-18T00:12:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:12:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:12:49.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-18T00:12:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:49.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:49.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:12:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:12:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 2cb77aa7-e9ed-4631-ae68-2c46fe019d9f 2024-09-18T00:12:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:12:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:12:49.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47' 2024-09-18T00:12:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:12:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:12:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:12:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-18T00:12:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:12:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:12:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-18T00:12:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:12:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 45ef8e7e-a26e-485b-868f-8d394feac98d 2024-09-18T00:12:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:12:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47' 2024-09-18T00:12:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:12:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:49.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:12:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:12:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:49.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47' 2024-09-18T00:12:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-18T00:12:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:12:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:12:49.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:12:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:49.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:49.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:12:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-18T00:12:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:12:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:12:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:49.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-18T00:12:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:12:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:12:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-18T00:12:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47' 2024-09-18T00:12:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:12:49.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:12:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-18T00:12:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:12:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:12:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-18T00:12:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47' 2024-09-18T00:12:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:12:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:12:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-18T00:12:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:12:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:12:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-18T00:12:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47' 2024-09-18T00:12:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:49.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:12:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:12:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-18T00:12:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:12:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:12:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 74b61702-1dd3-4fc5-a2c4-d9fdffc235df 2024-09-18T00:12:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:12:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:49.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:49.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:12:49.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:49.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:12:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:12:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:49.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47' 2024-09-18T00:12:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:49.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:12:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:12:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-18T00:12:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:12:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47' 2024-09-18T00:12:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:12:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:12:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-18T00:12:49.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:49.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:12:49.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:49.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:49.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:49.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:12:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:12:47' 2024-09-18T00:12:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:12:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:12:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:12:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-18T00:12:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:12:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:12:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:12:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:12:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:12:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:12:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:12:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:12:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:12:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-18T00:12:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:12:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:12:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-18T00:12:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:12:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:12:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-18T00:12:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:12:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:12:49.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-18T00:12:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:12:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:12:49.539 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-18T00:12:49.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd0 2024-09-18T00:12:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-18T00:12:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:49.642 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd1 2024-09-18T00:12:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-18T00:12:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd2 2024-09-18T00:12:49.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-18T00:12:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd3 2024-09-18T00:12:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-18T00:12:49.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:49.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd4 2024-09-18T00:12:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-18T00:12:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-18T00:12:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:50.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:12:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:50.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-18T00:12:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:12:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 92d2ac4c-65b0-4354-847f-ae86007ed627 2024-09-18T00:12:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:12:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:12:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:50.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:12:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-18T00:12:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:12:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-18T00:12:50.422 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-18T00:12:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:12:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-18T00:12:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-18T00:12:50.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:12:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:50.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-18T00:12:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:12:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 218e33a2-8ff3-48a3-a4d1-2243247be64d 2024-09-18T00:12:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:12:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:50.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:12:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:50.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:50.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:12:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-18T00:12:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:12:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:12:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:12:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-18T00:12:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47' 2024-09-18T00:12:50.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:12:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:50.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:12:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:12:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 45ef8e7e-a26e-485b-868f-8d394feac98d 2024-09-18T00:12:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:12:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:12:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 2cb77aa7-e9ed-4631-ae68-2c46fe019d9f 2024-09-18T00:12:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:12:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:12:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:12:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-18T00:12:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:12:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:12:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-18T00:12:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:12:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:50.579 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:12:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:12:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 74b61702-1dd3-4fc5-a2c4-d9fdffc235df 2024-09-18T00:12:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:12:50.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47 2024-09-18T00:12:50.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:12:50.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:12:50.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:12:50.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:12:50.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:12:47' 2024-09-18T00:12:50.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:12:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:12:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:12:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:12:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:12:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:12:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:12:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:12:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:12:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:12:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-18 00:12:47' 2024-09-18T00:12:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:12:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:12:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:12:47' 2024-09-18T00:12:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:12:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:12:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:12:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47' 2024-09-18T00:12:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:12:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:12:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-18T00:12:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:12:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:12:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-18T00:12:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47' 2024-09-18T00:12:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:12:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:12:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47' 2024-09-18T00:12:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:12:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-18T00:12:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:12:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:12:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-18T00:12:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:50.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:12:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-18T00:12:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:12:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:12:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-18T00:12:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:12:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:12:47' 2024-09-18T00:12:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:12:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:12:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:12:47' 2024-09-18T00:12:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:12:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:12:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:12:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:12:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:12:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:12:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-18T00:12:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:12:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:12:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:12:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:12:50.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-18T00:12:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:12:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:12:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:12:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:12:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:12:51.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:12:51.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-18T00:12:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:12:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:12:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:12:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-18T00:12:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:12:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:12:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:12:51.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:12:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-18T00:12:51.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-18T00:12:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:12:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:12:51.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-18T00:12:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd5 2024-09-18T00:12:51.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-18T00:12:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd6 2024-09-18T00:12:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd7 2024-09-18T00:12:51.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-18T00:12:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-18T00:12:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd8 2024-09-18T00:12:51.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-18T00:12:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd9 2024-09-18T00:12:51.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-18T00:12:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:12:53.740 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:12:53.737+0000 7f0114255640 -1 reset not still connected to 0x557d0a065e10 2024-09-18T00:12:53.741 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:12:53.737+0000 7f0114255640 -1 reset not still connected to 0x557d0b465110 2024-09-18T00:12:53.741 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:12:53.737+0000 7f0114255640 -1 reset not still connected to 0x557d2286c340 2024-09-18T00:12:53.741 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:12:53.737+0000 7f0114255640 -1 reset not still connected to 0x557d2286c820 2024-09-18T00:13:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=fa0fbb6df9f356af18c880b8b062885f 2024-09-18T00:13:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-18T00:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ fa0fbb6df9f356af18c880b8b062885f != \f\a\0\f\b\b\6\d\f\9\f\3\5\6\a\f\1\8\c\8\8\0\b\8\b\0\6\2\8\8\5\f ]] 2024-09-18T00:13:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=450d6a56e3992d09f637b15c9220ac0a 2024-09-18T00:13:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-18T00:13:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 450d6a56e3992d09f637b15c9220ac0a != \4\5\0\d\6\a\5\6\e\3\9\9\2\d\0\9\f\6\3\7\b\1\5\c\9\2\2\0\a\c\0\a ]] 2024-09-18T00:13:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:13:20.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 103110 2024-09-18T00:13:21.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=1b2a5ec7adddad4dcddbaa2057baa597 2024-09-18T00:13:21.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-18T00:13:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 1b2a5ec7adddad4dcddbaa2057baa597 != \1\b\2\a\5\e\c\7\a\d\d\d\a\d\4\d\c\d\d\b\a\a\2\0\5\7\b\a\a\5\9\7 ]] 2024-09-18T00:13:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=74f6b725c035779624a9523491a58281 2024-09-18T00:13:21.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-18T00:13:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 74f6b725c035779624a9523491a58281 != \7\4\f\6\b\7\2\5\c\0\3\5\7\7\9\6\2\4\a\9\5\2\3\4\9\1\a\5\8\2\8\1 ]] 2024-09-18T00:13:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=348bf5ba5c4e2ff1a61446075e5eb795 2024-09-18T00:13:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-18T00:13:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 348bf5ba5c4e2ff1a61446075e5eb795 != \3\4\8\b\f\5\b\a\5\c\4\e\2\f\f\1\a\6\1\4\4\6\0\7\5\e\5\e\b\7\9\5 ]] 2024-09-18T00:13:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=7696752a39825087a61f37dc7bec9dbd 2024-09-18T00:13:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-18T00:13:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 7696752a39825087a61f37dc7bec9dbd != \7\6\9\6\7\5\2\a\3\9\8\2\5\0\8\7\a\6\1\f\3\7\d\c\7\b\e\c\9\d\b\d ]] 2024-09-18T00:13:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:13:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 103111 2024-09-18T00:13:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=1811bf9394c11175b9dfdbdfbd03eb6a 2024-09-18T00:13:23.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-18T00:13:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 1811bf9394c11175b9dfdbdfbd03eb6a != \1\8\1\1\b\f\9\3\9\4\c\1\1\1\7\5\b\9\d\f\d\b\d\f\b\d\0\3\e\b\6\a ]] 2024-09-18T00:13:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=3026604cb5901059dbc3227982b962c9 2024-09-18T00:13:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-18T00:13:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 3026604cb5901059dbc3227982b962c9 != \3\0\2\6\6\0\4\c\b\5\9\0\1\0\5\9\d\b\c\3\2\2\7\9\8\2\b\9\6\2\c\9 ]] 2024-09-18T00:13:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=3d328b5784c5973b7bf9bd82b90d2460 2024-09-18T00:13:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-18T00:13:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 3d328b5784c5973b7bf9bd82b90d2460 != \3\d\3\2\8\b\5\7\8\4\c\5\9\7\3\b\7\b\f\9\b\d\8\2\b\9\0\d\2\4\6\0 ]] 2024-09-18T00:13:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:13:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 103114 2024-09-18T00:13:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:13:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 103116 2024-09-18T00:13:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:13:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 103118 2024-09-18T00:13:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:13:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 103120 2024-09-18T00:13:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:13:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 103122 2024-09-18T00:13:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=c557fd6d3e614c6f46130e7fe7fa64e5 2024-09-18T00:13:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-18T00:13:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ c557fd6d3e614c6f46130e7fe7fa64e5 != \c\5\5\7\f\d\6\d\3\e\6\1\4\c\6\f\4\6\1\3\0\e\7\f\e\7\f\a\6\4\e\5 ]] 2024-09-18T00:13:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:13:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 103124 2024-09-18T00:13:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:13:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 103125 2024-09-18T00:13:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:13:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-18T00:13:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-18T00:13:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:13:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:13:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-18T00:13:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:13:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:13:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-18T00:13:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:13:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:13:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:13:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: b89802c1-ca8a-4855-9327-8d4f21083a07 2024-09-18T00:13:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:13:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3147980.8,"bytes_per_snapshot":62959616.0,"last_snapshot_bytes":62959616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"} 2024-09-18T00:13:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:13:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:17 2024-09-18T00:13:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:13:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:13:24.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:13:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:13:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:22' 2024-09-18T00:13:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:13:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:13:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:13:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3147980.8,"bytes_per_snapshot":62959616.0,"last_snapshot_bytes":62959616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"} 2024-09-18T00:13:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:13:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:13:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:13:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:13:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:13:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:13:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-18 00:13:17' 2024-09-18T00:13:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:24.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:13:24.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:24.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:13:17' 2024-09-18T00:13:24.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:24.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:24.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:13:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1b89802c1-ca8a-4855-9327-8d4f21083a07up+replayingreplaying, {"bytes_per_second":3147980.8,"bytes_per_snapshot":62959616.0,"last_snapshot_bytes":62959616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:22image-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+replayingreplaying, {"bytes_per_second":3071795.2,"bytes_per_snapshot":61435904.0,"last_snapshot_bytes":61435904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:22image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+replayingreplaying, {"bytes_per_second":3224780.8,"bytes_per_snapshot":64495616.0,"last_snapshot_bytes":64495616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:22image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+replayingreplaying, {"bytes_per_second":3279462.4,"bytes_per_snapshot":65589248.0,"last_snapshot_bytes":65589248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:22image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+replayingreplaying, {"bytes_per_second":3317145.6,"bytes_per_snapshot":66342912.0,"last_snapshot_bytes":66342912,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:22image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+replayingreplaying, {"bytes_per_second":3229286.4,"bytes_per_snapshot":64585728.0,"last_snapshot_bytes":64585728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:22image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+replayingreplaying, {"bytes_per_second":3241984.0,"bytes_per_snapshot":64839680.0,"last_snapshot_bytes":64839680,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:22image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+replayingreplaying, {"bytes_per_second":3321036.8,"bytes_per_snapshot":66420736.0,"last_snapshot_bytes":66420736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:22image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+replayingreplaying, {"bytes_per_second":3227238.4,"bytes_per_snapshot":64544768.0,"last_snapshot_bytes":64544768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:22image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+replayingreplaying, {"bytes_per_second":3326361.6,"bytes_per_snapshot":66527232.0,"last_snapshot_bytes":66527232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:22' 2024-09-18T00:13:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:13:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:13:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:13:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:13:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:13:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-18T00:13:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:13:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:13:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-18T00:13:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-18T00:13:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:13:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-18T00:13:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:13:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:13:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary1 2024-09-18T00:13:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary1 2024-09-18T00:13:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-18T00:13:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-18T00:13:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:13:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:13:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-18T00:13:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:13:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:13:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-18T00:13:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:13:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:13:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:13:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 218e33a2-8ff3-48a3-a4d1-2243247be64d 2024-09-18T00:13:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:13:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3224780.8,"bytes_per_snapshot":64495616.0,"last_snapshot_bytes":64495616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"} 2024-09-18T00:13:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:13:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:17 2024-09-18T00:13:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:13:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:13:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:13:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:13:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:27' 2024-09-18T00:13:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:13:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:13:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:13:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3224780.8,"bytes_per_snapshot":64495616.0,"last_snapshot_bytes":64495616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"} 2024-09-18T00:13:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:13:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:13:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:13:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:13:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:13:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:13:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-18 00:13:17' 2024-09-18T00:13:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:13:28.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:13:17' 2024-09-18T00:13:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:13:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK942634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+replayingreplaying, {"bytes_per_second":3071795.2,"bytes_per_snapshot":61435904.0,"last_snapshot_bytes":61435904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27image-primary2218e33a2-8ff3-48a3-a4d1-2243247be64dup+replayingreplaying, {"bytes_per_second":3224780.8,"bytes_per_snapshot":64495616.0,"last_snapshot_bytes":64495616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+replayingreplaying, {"bytes_per_second":3279462.4,"bytes_per_snapshot":65589248.0,"last_snapshot_bytes":65589248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+replayingreplaying, {"bytes_per_second":3317145.6,"bytes_per_snapshot":66342912.0,"last_snapshot_bytes":66342912,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+replayingreplaying, {"bytes_per_second":3229286.4,"bytes_per_snapshot":64585728.0,"last_snapshot_bytes":64585728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+replayingreplaying, {"bytes_per_second":3241984.0,"bytes_per_snapshot":64839680.0,"last_snapshot_bytes":64839680,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+replayingreplaying, {"bytes_per_second":3321036.8,"bytes_per_snapshot":66420736.0,"last_snapshot_bytes":66420736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+replayingreplaying, {"bytes_per_second":3227238.4,"bytes_per_snapshot":64544768.0,"last_snapshot_bytes":64544768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+replayingreplaying, {"bytes_per_second":3326361.6,"bytes_per_snapshot":66527232.0,"last_snapshot_bytes":66527232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27' 2024-09-18T00:13:28.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:13:28.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:13:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:13:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:13:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:13:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-18T00:13:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:13:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:28.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:13:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-18T00:13:28.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-18T00:13:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:13:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-18T00:13:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:13:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:13:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary2 2024-09-18T00:13:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary2 2024-09-18T00:13:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-18T00:13:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-18T00:13:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:13:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:13:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-18T00:13:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:13:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:13:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-18T00:13:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:13:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:13:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:13:31.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 2cb77aa7-e9ed-4631-ae68-2c46fe019d9f 2024-09-18T00:13:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:13:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3279462.4,"bytes_per_snapshot":65589248.0,"last_snapshot_bytes":65589248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"} 2024-09-18T00:13:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:13:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:17 2024-09-18T00:13:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:13:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:13:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:13:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:13:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:27' 2024-09-18T00:13:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:13:31.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:13:31.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:13:31.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3279462.4,"bytes_per_snapshot":65589248.0,"last_snapshot_bytes":65589248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"} 2024-09-18T00:13:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:13:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:31.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:13:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:13:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:13:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:13:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:13:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-18 00:13:17' 2024-09-18T00:13:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:13:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:13:17' 2024-09-18T00:13:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:13:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK842634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+replayingreplaying, {"bytes_per_second":3071795.2,"bytes_per_snapshot":61435904.0,"last_snapshot_bytes":61435904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27image-primary32cb77aa7-e9ed-4631-ae68-2c46fe019d9fup+replayingreplaying, {"bytes_per_second":3279462.4,"bytes_per_snapshot":65589248.0,"last_snapshot_bytes":65589248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+replayingreplaying, {"bytes_per_second":3317145.6,"bytes_per_snapshot":66342912.0,"last_snapshot_bytes":66342912,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+replayingreplaying, {"bytes_per_second":3229286.4,"bytes_per_snapshot":64585728.0,"last_snapshot_bytes":64585728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+replayingreplaying, {"bytes_per_second":3241984.0,"bytes_per_snapshot":64839680.0,"last_snapshot_bytes":64839680,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+replayingreplaying, {"bytes_per_second":3321036.8,"bytes_per_snapshot":66420736.0,"last_snapshot_bytes":66420736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+replayingreplaying, {"bytes_per_second":3227238.4,"bytes_per_snapshot":64544768.0,"last_snapshot_bytes":64544768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+replayingreplaying, {"bytes_per_second":3326361.6,"bytes_per_snapshot":66527232.0,"last_snapshot_bytes":66527232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:27' 2024-09-18T00:13:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:13:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:13:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:13:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:13:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:13:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-18T00:13:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:13:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:13:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-18T00:13:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-18T00:13:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:13:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-18T00:13:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:13:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:13:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary3 2024-09-18T00:13:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary3 2024-09-18T00:13:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-18T00:13:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-18T00:13:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:13:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:13:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-18T00:13:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:13:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:13:33.933 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:13:33.930+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ea270 2024-09-18T00:13:33.933 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:13:33.930+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:13:33.933 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:13:33.930+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-18T00:13:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-18T00:13:34.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:13:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-18T00:13:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:13:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: b5b69d0c-e317-47f0-89bc-b0d989333a73 2024-09-18T00:13:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:13:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3317145.6,"bytes_per_snapshot":66342912.0,"last_snapshot_bytes":66342912,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"} 2024-09-18T00:13:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:13:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:17 2024-09-18T00:13:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:13:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:13:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:13:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:13:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:32' 2024-09-18T00:13:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:13:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:13:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:13:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3317145.6,"bytes_per_snapshot":66342912.0,"last_snapshot_bytes":66342912,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"} 2024-09-18T00:13:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:13:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:13:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:13:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:13:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:13:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:13:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-18 00:13:17' 2024-09-18T00:13:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:13:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:13:17' 2024-09-18T00:13:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:13:34.248 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:13:34.246+0000 7f9bfab49640 -1 reset not still connected to 0x55cd37c20d00 2024-09-18T00:13:34.248 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:13:34.246+0000 7f9bfab49640 -1 reset not still connected to 0x55cd452c9930 2024-09-18T00:13:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK742634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+replayingreplaying, {"bytes_per_second":3071795.2,"bytes_per_snapshot":61435904.0,"last_snapshot_bytes":61435904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:32image-primary4b5b69d0c-e317-47f0-89bc-b0d989333a73up+replayingreplaying, {"bytes_per_second":3317145.6,"bytes_per_snapshot":66342912.0,"last_snapshot_bytes":66342912,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:32image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+replayingreplaying, {"bytes_per_second":3229286.4,"bytes_per_snapshot":64585728.0,"last_snapshot_bytes":64585728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:32image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+replayingreplaying, {"bytes_per_second":3241984.0,"bytes_per_snapshot":64839680.0,"last_snapshot_bytes":64839680,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:32image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+replayingreplaying, {"bytes_per_second":3321036.8,"bytes_per_snapshot":66420736.0,"last_snapshot_bytes":66420736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:32image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+replayingreplaying, {"bytes_per_second":3227238.4,"bytes_per_snapshot":64544768.0,"last_snapshot_bytes":64544768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:32image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+replayingreplaying, {"bytes_per_second":3326361.6,"bytes_per_snapshot":66527232.0,"last_snapshot_bytes":66527232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:32' 2024-09-18T00:13:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:13:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:13:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:13:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:13:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:34.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:13:34.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-18T00:13:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:13:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:13:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-18T00:13:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-18T00:13:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:13:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-18T00:13:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:13:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:13:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary4 2024-09-18T00:13:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary4 2024-09-18T00:13:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-18T00:13:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-18T00:13:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:13:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:13:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-18T00:13:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:13:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:13:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-18T00:13:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:13:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-18T00:13:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:13:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 630658f3-635d-4f74-99f3-bdef1657b63c 2024-09-18T00:13:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:13:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3229286.4,"bytes_per_snapshot":64585728.0,"last_snapshot_bytes":64585728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"} 2024-09-18T00:13:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:13:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:17 2024-09-18T00:13:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:13:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:13:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:13:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:13:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:32' 2024-09-18T00:13:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:13:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:13:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:13:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3229286.4,"bytes_per_snapshot":64585728.0,"last_snapshot_bytes":64585728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"} 2024-09-18T00:13:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:13:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:13:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:13:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:13:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:13:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:13:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-18 00:13:17' 2024-09-18T00:13:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:13:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:13:17' 2024-09-18T00:13:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:13:37.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK642634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+replayingreplaying, {"bytes_per_second":3071795.2,"bytes_per_snapshot":61435904.0,"last_snapshot_bytes":61435904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:32image-primary5630658f3-635d-4f74-99f3-bdef1657b63cup+replayingreplaying, {"bytes_per_second":3229286.4,"bytes_per_snapshot":64585728.0,"last_snapshot_bytes":64585728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:32image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+replayingreplaying, {"bytes_per_second":3241984.0,"bytes_per_snapshot":64839680.0,"last_snapshot_bytes":64839680,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:32image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+replayingreplaying, {"bytes_per_second":3321036.8,"bytes_per_snapshot":66420736.0,"last_snapshot_bytes":66420736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:32image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+replayingreplaying, {"bytes_per_second":3227238.4,"bytes_per_snapshot":64544768.0,"last_snapshot_bytes":64544768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:32image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+replayingreplaying, {"bytes_per_second":3326361.6,"bytes_per_snapshot":66527232.0,"last_snapshot_bytes":66527232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:32' 2024-09-18T00:13:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:13:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:13:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:13:37.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:13:37.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:37.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:13:37.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-18T00:13:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:13:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:13:37.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-18T00:13:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-18T00:13:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:13:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-18T00:13:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:13:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:13:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary5 2024-09-18T00:13:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary5 2024-09-18T00:13:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-18T00:13:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-18T00:13:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:13:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:13:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-18T00:13:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:13:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:13:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-18T00:13:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:13:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:13:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:13:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 92d2ac4c-65b0-4354-847f-ae86007ed627 2024-09-18T00:13:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:13:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3241984.0,"bytes_per_snapshot":64839680.0,"last_snapshot_bytes":64839680,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"} 2024-09-18T00:13:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:13:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:17 2024-09-18T00:13:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:13:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:13:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:13:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:13:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:37' 2024-09-18T00:13:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:13:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:13:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:13:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3241984.0,"bytes_per_snapshot":64839680.0,"last_snapshot_bytes":64839680,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"} 2024-09-18T00:13:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:13:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:13:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:13:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:13:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:13:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:13:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-18 00:13:17' 2024-09-18T00:13:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:40.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:13:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:13:17' 2024-09-18T00:13:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:40.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:13:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK542634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+replayingreplaying, {"bytes_per_second":3071795.2,"bytes_per_snapshot":61435904.0,"last_snapshot_bytes":61435904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:37image-primary692d2ac4c-65b0-4354-847f-ae86007ed627up+replayingreplaying, {"bytes_per_second":3241984.0,"bytes_per_snapshot":64839680.0,"last_snapshot_bytes":64839680,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:37image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+replayingreplaying, {"bytes_per_second":3321036.8,"bytes_per_snapshot":66420736.0,"last_snapshot_bytes":66420736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:37image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+replayingreplaying, {"bytes_per_second":3227238.4,"bytes_per_snapshot":64544768.0,"last_snapshot_bytes":64544768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:37image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+replayingreplaying, {"bytes_per_second":3326361.6,"bytes_per_snapshot":66527232.0,"last_snapshot_bytes":66527232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:37' 2024-09-18T00:13:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:13:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:13:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:13:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:13:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:13:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-18T00:13:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:13:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:13:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-18T00:13:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-18T00:13:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:13:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-18T00:13:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:41.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:13:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:13:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary6 2024-09-18T00:13:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary6 2024-09-18T00:13:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-18T00:13:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-18T00:13:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:13:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:13:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-18T00:13:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:13:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:13:43.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-18T00:13:43.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:43.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:43.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:13:43.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:43.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:13:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:13:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 74b61702-1dd3-4fc5-a2c4-d9fdffc235df 2024-09-18T00:13:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:13:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3321036.8,"bytes_per_snapshot":66420736.0,"last_snapshot_bytes":66420736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"} 2024-09-18T00:13:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:13:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:17 2024-09-18T00:13:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:13:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:13:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:13:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:13:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:42' 2024-09-18T00:13:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:13:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:13:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:13:43.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3321036.8,"bytes_per_snapshot":66420736.0,"last_snapshot_bytes":66420736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"} 2024-09-18T00:13:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:13:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:13:43.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:13:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:13:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:13:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:13:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-18 00:13:17' 2024-09-18T00:13:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:13:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:13:17' 2024-09-18T00:13:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:13:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+replayingreplaying, {"bytes_per_second":3071795.2,"bytes_per_snapshot":61435904.0,"last_snapshot_bytes":61435904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:42image-primary774b61702-1dd3-4fc5-a2c4-d9fdffc235dfup+replayingreplaying, {"bytes_per_second":3321036.8,"bytes_per_snapshot":66420736.0,"last_snapshot_bytes":66420736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:42image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+replayingreplaying, {"bytes_per_second":3227238.4,"bytes_per_snapshot":64544768.0,"last_snapshot_bytes":64544768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:42image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+replayingreplaying, {"bytes_per_second":3326361.6,"bytes_per_snapshot":66527232.0,"last_snapshot_bytes":66527232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:42' 2024-09-18T00:13:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:13:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:13:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:13:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:13:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:13:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:43.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:43.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:43.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-18T00:13:43.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:13:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:13:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-18T00:13:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-18T00:13:43.401 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:13:43.399+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782eea0 2024-09-18T00:13:43.401 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:13:43.399+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-18T00:13:43.401 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:13:43.399+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1abbeea0 2024-09-18T00:13:44.707 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:13:44.705+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-18T00:13:44.710 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:13:44.708+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0cb6e0d0 2024-09-18T00:13:44.710 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:13:44.708+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1abbeea0 2024-09-18T00:13:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:13:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-18T00:13:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:13:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:13:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary7 2024-09-18T00:13:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary7 2024-09-18T00:13:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-18T00:13:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-18T00:13:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:13:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:13:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-18T00:13:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:13:45.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:13:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-18T00:13:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:13:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:13:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:13:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 45ef8e7e-a26e-485b-868f-8d394feac98d 2024-09-18T00:13:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:13:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3227238.4,"bytes_per_snapshot":64544768.0,"last_snapshot_bytes":64544768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"} 2024-09-18T00:13:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:13:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:17 2024-09-18T00:13:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:13:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:13:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:13:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:13:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:42' 2024-09-18T00:13:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:13:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:13:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:13:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3227238.4,"bytes_per_snapshot":64544768.0,"last_snapshot_bytes":64544768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"} 2024-09-18T00:13:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:13:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:13:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:46.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:13:46.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:13:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:13:46.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:13:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-18 00:13:17' 2024-09-18T00:13:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:13:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:13:17' 2024-09-18T00:13:46.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:13:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK342634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+replayingreplaying, {"bytes_per_second":3071795.2,"bytes_per_snapshot":61435904.0,"last_snapshot_bytes":61435904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:42image-primary845ef8e7e-a26e-485b-868f-8d394feac98dup+replayingreplaying, {"bytes_per_second":3227238.4,"bytes_per_snapshot":64544768.0,"last_snapshot_bytes":64544768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618371,"remote_snapshot_timestamp":1726618371,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:42image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+replayingreplaying, {"bytes_per_second":3326361.6,"bytes_per_snapshot":66527232.0,"last_snapshot_bytes":66527232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:42' 2024-09-18T00:13:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:13:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:13:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:17' 2024-09-18T00:13:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:13:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:13:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:13:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-18T00:13:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:13:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:13:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-18T00:13:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-18T00:13:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:13:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-18T00:13:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:13:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:13:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary8 2024-09-18T00:13:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary8 2024-09-18T00:13:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-18T00:13:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-18T00:13:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:13:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:13:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-18T00:13:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:13:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:13:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-18T00:13:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:13:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:13:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:13:49.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: b7f435e7-f034-4987-af7c-ed84c4d984bf 2024-09-18T00:13:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:13:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66527232.0,"last_snapshot_bytes":66527232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"} 2024-09-18T00:13:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:13:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:47 2024-09-18T00:13:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:13:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:13:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:13:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:13:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:47' 2024-09-18T00:13:49.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:13:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:13:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:13:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66527232.0,"last_snapshot_bytes":66527232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"} 2024-09-18T00:13:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:13:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:13:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:13:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:13:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:13:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:13:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:47' 2024-09-18T00:13:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-18 00:13:47' 2024-09-18T00:13:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:13:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:13:47' 2024-09-18T00:13:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:13:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK242634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61435904.0,"last_snapshot_bytes":61435904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:47image-primary9b7f435e7-f034-4987-af7c-ed84c4d984bfup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66527232.0,"last_snapshot_bytes":66527232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:47' 2024-09-18T00:13:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:13:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:13:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:47' 2024-09-18T00:13:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:13:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:13:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:13:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-18T00:13:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:13:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:13:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-18T00:13:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-18T00:13:50.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:13:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-18T00:13:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:13:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:13:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary9 2024-09-18T00:13:51.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary9 2024-09-18T00:13:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-18T00:13:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-18T00:13:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:13:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:13:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-18T00:13:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:13:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:13:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-18T00:13:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:13:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:13:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:13:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:13:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:13:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 7d207094-b0f0-4686-a9cf-144f851a07f8 2024-09-18T00:13:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:13:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61435904.0,"last_snapshot_bytes":61435904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"} 2024-09-18T00:13:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:13:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:47 2024-09-18T00:13:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:13:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:13:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:13:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:13:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:13:47' 2024-09-18T00:13:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:13:52.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:13:52.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:13:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61435904.0,"last_snapshot_bytes":61435904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"} 2024-09-18T00:13:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:13:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:13:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:13:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:13:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:13:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:13:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:47' 2024-09-18T00:13:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-18 00:13:47' 2024-09-18T00:13:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:13:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:13:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:13:47' 2024-09-18T00:13:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:13:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:13:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:13:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary107d207094-b0f0-4686-a9cf-144f851a07f8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61435904.0,"last_snapshot_bytes":61435904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618369,"remote_snapshot_timestamp":1726618369,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:13:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:13:47' 2024-09-18T00:13:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:13:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:13:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:13:47' 2024-09-18T00:13:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:13:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:13:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:13:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:13:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:13:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-18T00:13:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:13:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:13:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-18T00:13:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-18T00:13:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:13:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-18T00:13:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:13:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:13:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:13:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary10 2024-09-18T00:13:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary10 2024-09-18T00:13:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..10} 2024-09-18T00:13:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS=() 2024-09-18T00:13:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS=() 2024-09-18T00:13:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS=() 2024-09-18T00:13:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS=() 2024-09-18T00:13:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ RET=0 2024-09-18T00:13:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-18T00:13:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary1 2024-09-18T00:13:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-18T00:13:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:13:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:13:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:13:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-18T00:13:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:13:54.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:13:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:13:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:13:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-18T00:13:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:13:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:13:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-18T00:13:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:13:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:13:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-18T00:13:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:13:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-18T00:13:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary1': 2024-09-18T00:13:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:13:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:13:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:13:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 2bfb13929202 2024-09-18T00:13:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.2bfb13929202 2024-09-18T00:13:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:13:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:13:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:13:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:13:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:13:54 2024 2024-09-18T00:13:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:13:54 2024 2024-09-18T00:13:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:13:54 2024 2024-09-18T00:13:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:13:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:13:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4 2024-09-18T00:13:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:13:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:13:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:13:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-18T00:13:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd0 2024-09-18T00:13:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:13:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-18T00:13:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:13:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:13:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:13:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 4c4f2c16-b2ce-41c3-b2aa-9e355b8f9748 2024-09-18T00:13:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:13:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:13:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:13:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:13:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:13:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:13:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:13:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:13:55.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary1 2024-09-18T00:13:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-18T00:13:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:13:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary1 2024-09-18T00:13:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:13:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-18T00:13:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:13:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:13:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:13:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:13:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary1 2024-09-18T00:13:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:13:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-18T00:13:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary2 2024-09-18T00:13:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:13:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-18T00:13:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary1 2024-09-18T00:13:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-18T00:13:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:13:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:13:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:13:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-18T00:13:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:13:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:13:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:13:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:13:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-18T00:13:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:13:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:13:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-18T00:13:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:13:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:13:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-18T00:13:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:13:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-18T00:13:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:13:56.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1986 2024-09-18T00:13:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-18T00:13:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:13:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary2': 2024-09-18T00:13:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:13:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:13:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:13:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 2c0b22dbb441 2024-09-18T00:13:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.2c0b22dbb441 2024-09-18T00:13:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:13:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:13:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:13:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:13:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:13:55 2024 2024-09-18T00:13:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:13:55 2024 2024-09-18T00:13:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:13:55 2024 2024-09-18T00:13:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:13:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:13:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: d64cf4b9-98c4-4f70-a161-76e8c07ce938 2024-09-18T00:13:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:13:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:13:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:13:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-18T00:13:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd1 2024-09-18T00:13:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:13:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-18T00:13:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:13:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:13:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:13:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: af4786b8-7d16-4328-bd2d-99e6c0035524 2024-09-18T00:13:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:13:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:13:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:13:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:13:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:13:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:13:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:13:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:13:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary2 2024-09-18T00:13:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-18T00:13:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:13:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary2 2024-09-18T00:13:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:13:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-18T00:13:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:13:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:13:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:13:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:13:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:13:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:13:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary2 2024-09-18T00:13:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-18T00:13:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary3 2024-09-18T00:13:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-18T00:13:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary2 2024-09-18T00:13:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-18T00:13:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:13:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:13:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:13:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-18T00:13:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:13:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:13:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:13:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:13:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-18T00:13:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:13:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:13:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-18T00:13:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:57.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:13:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:13:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-18T00:13:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:13:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-18T00:13:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:13:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-18T00:13:57.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1988 2024-09-18T00:13:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:13:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary3': 2024-09-18T00:13:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:13:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:13:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:13:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 2c1a8c371d4d 2024-09-18T00:13:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.2c1a8c371d4d 2024-09-18T00:13:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:13:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:13:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:13:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:13:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:13:56 2024 2024-09-18T00:13:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:13:56 2024 2024-09-18T00:13:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:13:56 2024 2024-09-18T00:13:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:13:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:13:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 5aa63b87-d45e-4d8e-a667-5dafb6b4ef7c 2024-09-18T00:13:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:13:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:13:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:13:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-18T00:13:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd2 2024-09-18T00:13:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:13:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-18T00:13:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:13:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:13:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:13:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: b3839e0c-9cc1-4348-accb-17216a46b001 2024-09-18T00:13:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:13:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:13:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:13:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:13:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:13:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:13:58.023 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:13:58.020+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:13:58.023 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:13:58.020+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-18T00:13:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:13:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:13:58.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary3 2024-09-18T00:13:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-18T00:13:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:13:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary3 2024-09-18T00:13:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:13:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-18T00:13:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:13:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:13:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:13:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:13:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary3 2024-09-18T00:13:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-18T00:13:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:13:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary4 2024-09-18T00:13:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:13:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-18T00:13:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary3 2024-09-18T00:13:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-18T00:13:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:13:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:13:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:13:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-18T00:13:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:13:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:13:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:13:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:13:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-18T00:13:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:13:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:13:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-18T00:13:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:13:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:13:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-18T00:13:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:13:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-18T00:13:58.703 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:13:58.700+0000 7f9bfab49640 -1 reset not still connected to 0x55cd400eb520 2024-09-18T00:13:58.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:13:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1990 2024-09-18T00:13:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:13:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-18T00:13:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary4': 2024-09-18T00:13:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:13:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:13:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:13:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 2c27987f6d2a 2024-09-18T00:13:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.2c27987f6d2a 2024-09-18T00:13:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:13:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:13:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:13:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:13:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:13:58 2024 2024-09-18T00:13:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:13:58 2024 2024-09-18T00:13:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:13:58 2024 2024-09-18T00:13:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:13:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:13:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 98347238-de9a-416e-80a1-1bb76d54ed82 2024-09-18T00:13:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:13:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:13:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:13:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-18T00:13:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd3 2024-09-18T00:13:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:13:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-18T00:13:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:13:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:13:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:13:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 5d2071be-9e0a-4d3e-b5f2-b875c7000d11 2024-09-18T00:13:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:13:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:13:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:13:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:13:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:13:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:13:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:13:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:13:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary4 2024-09-18T00:13:59.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-18T00:13:59.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:13:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary4 2024-09-18T00:13:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:13:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-18T00:13:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:13:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:13:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:13:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:13:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:13:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-18T00:13:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary4 2024-09-18T00:13:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary5 2024-09-18T00:13:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:13:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-18T00:13:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:13:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:13:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary4 2024-09-18T00:13:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:13:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-18T00:13:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-18T00:13:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:13:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:13:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:13:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:13:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-18T00:13:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:13:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:13:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:13:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-18T00:13:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:13:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:13:59.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-18T00:13:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:13:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-18T00:13:59.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:13:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-18T00:13:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:13:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:13:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:13:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:13:59.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary5': 2024-09-18T00:13:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:13:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:13:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:13:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 2c357c576921 2024-09-18T00:13:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.2c357c576921 2024-09-18T00:13:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:13:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:13:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:13:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:13:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:13:59 2024 2024-09-18T00:13:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:13:59 2024 2024-09-18T00:13:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:13:59 2024 2024-09-18T00:13:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:13:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:13:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: eff149be-58c6-4f92-96d5-04d6dc49412e 2024-09-18T00:13:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:13:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:13:59.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:13:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-18T00:13:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd4 2024-09-18T00:13:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:13:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-18T00:13:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:13:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:13:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:13:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: eedaf3a4-3b08-4045-8ef2-a95dd0baebf2 2024-09-18T00:13:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:13:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:13:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:13:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:13:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:13:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:13:59.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 1992 2024-09-18T00:13:59.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:13:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:done 2024-09-18T00:13:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:13:59.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary5 2024-09-18T00:13:59.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-18T00:13:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:13:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary5 2024-09-18T00:13:59.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:13:59.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-18T00:13:59.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:13:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:13:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:13:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:13:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:13:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary5 2024-09-18T00:13:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-18T00:13:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary6 2024-09-18T00:13:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-18T00:13:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:13:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:13:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:13:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:13:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary5 2024-09-18T00:13:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:13:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-18T00:13:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-18T00:13:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:13:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:13:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:13:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:13:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:13:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:13:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:13:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-18T00:14:00.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:14:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:14:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:14:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-18T00:14:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:14:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:14:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-18T00:14:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:14:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-18T00:14:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:14:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1995 2024-09-18T00:14:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-18T00:14:00.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary6': 2024-09-18T00:14:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:14:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:14:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:14:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 2c44d4fef49c 2024-09-18T00:14:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.2c44d4fef49c 2024-09-18T00:14:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:14:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:14:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:14:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:14:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:14:00 2024 2024-09-18T00:14:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:14:00 2024 2024-09-18T00:14:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:14:00 2024 2024-09-18T00:14:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:14:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:14:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6 2024-09-18T00:14:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:14:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:14:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:14:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:14:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:14:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:14:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-18T00:14:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd5 2024-09-18T00:14:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:14:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-18T00:14:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:14:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:14:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:14:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 2544ac4c-2077-4f38-8de2-99e1875a4ff3 2024-09-18T00:14:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:14:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:14:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:14:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:14:00.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:14:00.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:14:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:14:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:14:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary6 2024-09-18T00:14:00.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-18T00:14:00.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:14:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary6 2024-09-18T00:14:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:14:00.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-18T00:14:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:14:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:14:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:14:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:14:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-18T00:14:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary6 2024-09-18T00:14:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary7 2024-09-18T00:14:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:00.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:14:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-18T00:14:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:14:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:14:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:14:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary6 2024-09-18T00:14:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-18T00:14:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-18T00:14:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:14:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:14:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:14:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-18T00:14:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:14:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:01.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:14:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:14:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-18T00:14:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:14:01.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-18T00:14:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:14:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-18T00:14:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1994 2024-09-18T00:14:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:14:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:14:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:14:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1998 2024-09-18T00:14:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:14:01.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-18T00:14:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary7': 2024-09-18T00:14:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:14:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:14:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:14:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 2c568af2b62e 2024-09-18T00:14:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.2c568af2b62e 2024-09-18T00:14:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:14:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:14:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:14:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:14:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:14:01 2024 2024-09-18T00:14:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:14:01 2024 2024-09-18T00:14:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:14:01 2024 2024-09-18T00:14:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:14:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:14:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 02b583ab-f4b1-4c02-b3ba-6dd65cd3316c 2024-09-18T00:14:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:14:01.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:14:01.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:14:01.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-18T00:14:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd6 2024-09-18T00:14:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:14:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-18T00:14:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: cc62c94c-d485-419a-bf54-7568fa09a4bf 2024-09-18T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:14:02.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:14:02.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:14:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:14:02.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:14:02.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary7 2024-09-18T00:14:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-18T00:14:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:14:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary7 2024-09-18T00:14:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:14:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-18T00:14:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:02.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:14:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:14:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:14:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary7 2024-09-18T00:14:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-18T00:14:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary8 2024-09-18T00:14:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-18T00:14:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:14:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary7 2024-09-18T00:14:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-18T00:14:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:14:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:14:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-18T00:14:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:14:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:14:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:14:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-18T00:14:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:14:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:14:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:14:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-18T00:14:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:14:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-18T00:14:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:14:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-18T00:14:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 1997 2024-09-18T00:14:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:14:02.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-18T00:14:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:14:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:14:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:14:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary8': 2024-09-18T00:14:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:14:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:14:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:14:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 2c67927611a4 2024-09-18T00:14:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.2c67927611a4 2024-09-18T00:14:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:14:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:14:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:14:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:14:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:14:02 2024 2024-09-18T00:14:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:14:02 2024 2024-09-18T00:14:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:14:02 2024 2024-09-18T00:14:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:14:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:14:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: a827873c-19b1-443f-b747-bed14c3f54c3 2024-09-18T00:14:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:14:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:14:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:14:02.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-18T00:14:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2001 2024-09-18T00:14:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd7 2024-09-18T00:14:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:14:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-18T00:14:03.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:14:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:14:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:14:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 12fff7f8-4564-453c-82ec-81cd92e0d4c6 2024-09-18T00:14:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:14:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:14:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:14:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:14:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:14:03.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:14:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:14:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:14:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary8 2024-09-18T00:14:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-18T00:14:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:14:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary8 2024-09-18T00:14:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:14:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-18T00:14:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:14:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:03.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:14:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:14:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-18T00:14:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary9 2024-09-18T00:14:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:14:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-18T00:14:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary8 2024-09-18T00:14:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:14:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:14:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-18T00:14:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary8 2024-09-18T00:14:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:14:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:14:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-18T00:14:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:14:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:03.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-18T00:14:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:14:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-18T00:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-18T00:14:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:14:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-18T00:14:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2000 2024-09-18T00:14:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:14:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:14:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:14:03.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2004 2024-09-18T00:14:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:14:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:03.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-18T00:14:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary9': 2024-09-18T00:14:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:14:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:14:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:14:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 2c778a3e3834 2024-09-18T00:14:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.2c778a3e3834 2024-09-18T00:14:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:14:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:14:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:14:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:14:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:14:03 2024 2024-09-18T00:14:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:14:03 2024 2024-09-18T00:14:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:14:03 2024 2024-09-18T00:14:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:14:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:14:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 814425c6-1326-454a-85d2-08f9f100d357 2024-09-18T00:14:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:14:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:14:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:14:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-18T00:14:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd8 2024-09-18T00:14:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:14:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-18T00:14:04.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:14:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:14:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:14:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: cf39429b-aed5-4000-8471-74eb0fdc9000 2024-09-18T00:14:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:14:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:14:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:14:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:14:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:14:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:14:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:14:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:14:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary9 2024-09-18T00:14:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-18T00:14:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:14:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary9 2024-09-18T00:14:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:14:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-18T00:14:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:14:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:14:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:14:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-18T00:14:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary9 2024-09-18T00:14:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary10 2024-09-18T00:14:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-18T00:14:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:14:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:14:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:14:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary9 2024-09-18T00:14:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-18T00:14:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-18T00:14:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:14:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:14:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:14:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:14:04.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-18T00:14:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:14:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:14:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:14:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-18T00:14:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:14:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-18T00:14:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:14:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-18T00:14:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2003 2024-09-18T00:14:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2008 2024-09-18T00:14:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:14:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:14:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:14:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:14:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:14:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:14:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:14:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-18T00:14:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary10': 2024-09-18T00:14:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:14:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:14:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:14:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 2c8812122c31 2024-09-18T00:14:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.2c8812122c31 2024-09-18T00:14:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:14:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:14:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:14:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:14:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:14:04 2024 2024-09-18T00:14:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:14:04 2024 2024-09-18T00:14:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:14:04 2024 2024-09-18T00:14:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:14:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:14:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: d2b68ce6-80c4-4597-8006-4885dadbe8d9 2024-09-18T00:14:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:14:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:14:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:14:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-18T00:14:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd9 2024-09-18T00:14:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:14:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-18T00:14:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:14:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:14:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:14:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 14765f37-8c13-48da-94aa-256380bccafe 2024-09-18T00:14:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:14:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:14:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:14:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:14:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:14:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:14:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary10 2024-09-18T00:14:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-18T00:14:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:14:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary10 2024-09-18T00:14:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:14:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-18T00:14:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:14:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:14:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 106760 2024-09-18T00:14:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:14:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary10 2024-09-18T00:14:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:14:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary10 2024-09-18T00:14:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-18T00:14:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:14:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-18T00:14:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2006 2024-09-18T00:14:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:14:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:14:05.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:14:05.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2011 2024-09-18T00:14:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:14:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:14:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2010 2024-09-18T00:14:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2009 2024-09-18T00:14:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:14:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:14:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:14:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:14:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:14:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:14:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:14:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:14:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2013 2024-09-18T00:14:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2012 2024-09-18T00:14:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:07.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:07.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:07.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:14:07.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:07.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:07.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:07.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:14:08.108 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:14:08.107+0000 7f6acfc5f640 -1 reset not still connected to 0x555c09ea0d00 2024-09-18T00:14:08.108 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:14:08.107+0000 7f6acfc5f640 -1 reset not still connected to 0x555c09ea1d40 2024-09-18T00:14:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2015 2024-09-18T00:14:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:14:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:14:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:14:08.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2014 2024-09-18T00:14:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:14:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:08.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:14:09.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2016 2024-09-18T00:14:09.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2017 2024-09-18T00:14:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:14:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:14:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:14:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:14:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:14:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:14:10.603 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:10.601+0000 7f9bfab49640 -1 reset not still connected to 0x55cd24742b60 2024-09-18T00:14:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2018 2024-09-18T00:14:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2019 2024-09-18T00:14:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:14:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:14:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:14:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:14:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:14:10.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:14:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:14:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:14:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:14:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:11.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:14:11.987 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:14:11.986+0000 7f706e974640 -1 reset not still connected to 0x55df471ed380 2024-09-18T00:14:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:14:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:14:12.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:14:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:14:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:12.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:14:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2023 2024-09-18T00:14:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2022 2024-09-18T00:14:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2021 2024-09-18T00:14:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2020 2024-09-18T00:14:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:12.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:13.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:13.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:14:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:14:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:14:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2024 2024-09-18T00:14:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2026 2024-09-18T00:14:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2027 2024-09-18T00:14:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2025 2024-09-18T00:14:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:14.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:14.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:14.215 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:14.204+0000 7f9bfab49640 -1 reset not still connected to 0x55cd22605380 2024-09-18T00:14:14.217 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:14.207+0000 7f9bfab49640 -1 reset not still connected to 0x55cd32ff76c0 2024-09-18T00:14:14.220 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:14.207+0000 7f9bfab49640 -1 reset not still connected to 0x55cd34265930 2024-09-18T00:14:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2028 2024-09-18T00:14:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2029 2024-09-18T00:14:15.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:14:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:14:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:14:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:14:15.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:14:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:14:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:14:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:14:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:14:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:14:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:14:16.878 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:14:16.876+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4dd0 2024-09-18T00:14:16.883 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:14:16.882+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ea270 2024-09-18T00:14:16.886 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:14:16.883+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:14:16.886 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:14:16.883+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-18T00:14:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2031 2024-09-18T00:14:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:14:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:14:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:14:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2033 2024-09-18T00:14:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:14:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:17.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:14:17.088 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:17.083+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-18T00:14:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:14:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:14:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:14:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:14:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:14:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:14:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:14:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2030 2024-09-18T00:14:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2032 2024-09-18T00:14:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:14:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:14:18.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2034 2024-09-18T00:14:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2036 2024-09-18T00:14:18.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2035 2024-09-18T00:14:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2038 2024-09-18T00:14:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2037 2024-09-18T00:14:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:18.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2039 2024-09-18T00:14:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:14:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:14:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:14:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:20.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:14:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:14:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:14:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:14:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:14:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:14:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:14:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:14:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2040 2024-09-18T00:14:21.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2041 2024-09-18T00:14:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:14:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:21.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:14:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:14:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:14:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:14:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:14:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:14:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:14:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:14:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:14:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:14:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:14:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:14:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:14:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:14:22.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2042 2024-09-18T00:14:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2043 2024-09-18T00:14:22.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2049 2024-09-18T00:14:23.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2048 2024-09-18T00:14:23.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2045 2024-09-18T00:14:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2044 2024-09-18T00:14:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2047 2024-09-18T00:14:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2046 2024-09-18T00:14:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:23.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:23.991 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:23.988+0000 7f9bfab49640 -1 reset not still connected to 0x55cd19c0e680 2024-09-18T00:14:23.991 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:23.988+0000 7f9bfab49640 -1 reset not still connected to 0x55cd22fd5c70 2024-09-18T00:14:23.991 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:23.988+0000 7f9bfab49640 -1 reset not still connected to 0x55cd46da9e10 2024-09-18T00:14:23.991 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:23.988+0000 7f9bfab49640 -1 reset not still connected to 0x55cd504f9450 2024-09-18T00:14:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:14:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:14:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:14:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:14:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:14:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:14:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:14:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:14:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:14:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:14:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2051 2024-09-18T00:14:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2050 2024-09-18T00:14:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:14:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:14:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:14:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:14:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:14:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:14:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:14:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:14:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:14:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:14:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:14:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:14:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:14:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:14:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:14:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2053 2024-09-18T00:14:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2052 2024-09-18T00:14:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2055 2024-09-18T00:14:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2059 2024-09-18T00:14:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2056 2024-09-18T00:14:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2057 2024-09-18T00:14:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2054 2024-09-18T00:14:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2058 2024-09-18T00:14:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:14:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:14:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:14:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:14:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:14:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:14:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:30.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:14:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:30.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:14:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:14:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:14:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:14:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2061 2024-09-18T00:14:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2060 2024-09-18T00:14:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:14:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:14:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:14:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:14:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:14:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:14:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:14:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:14:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:14:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:14:32.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:14:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:14:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:14:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:14:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:14:32.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2062 2024-09-18T00:14:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2063 2024-09-18T00:14:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:33.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2064 2024-09-18T00:14:33.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2068 2024-09-18T00:14:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2069 2024-09-18T00:14:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2065 2024-09-18T00:14:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2067 2024-09-18T00:14:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2066 2024-09-18T00:14:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:14:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:14:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:14:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:14:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:14:35.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:14:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:14:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:14:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:14:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:14:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:14:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2070 2024-09-18T00:14:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2071 2024-09-18T00:14:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:14:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:14:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:14:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:14:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:14:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:14:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:14:37.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:14:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:14:37.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:14:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:14:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:14:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:14:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:14:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:14:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2072 2024-09-18T00:14:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2073 2024-09-18T00:14:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2074 2024-09-18T00:14:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2077 2024-09-18T00:14:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2079 2024-09-18T00:14:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2076 2024-09-18T00:14:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2078 2024-09-18T00:14:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2075 2024-09-18T00:14:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:39.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:39.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:14:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:14:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:14:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:14:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:14:40.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:14:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:14:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:14:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:14:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:14:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:14:41.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2081 2024-09-18T00:14:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2080 2024-09-18T00:14:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:14:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:14:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:14:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:14:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:14:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:14:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:14:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:14:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:14:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:14:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:14:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:14:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:14:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:14:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:14:42.066 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:14:42.061+0000 7fc275b74640 -1 reset not still connected to 0x559cc83876c0 2024-09-18T00:14:42.066 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:14:42.062+0000 7fc275b74640 -1 reset not still connected to 0x559cc8a5a340 2024-09-18T00:14:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2082 2024-09-18T00:14:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2083 2024-09-18T00:14:43.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2085 2024-09-18T00:14:44.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2088 2024-09-18T00:14:44.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2084 2024-09-18T00:14:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2089 2024-09-18T00:14:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2086 2024-09-18T00:14:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2087 2024-09-18T00:14:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:14:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:14:45.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:14:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:14:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:14:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:14:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:14:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:14:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:14:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:14:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:14:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2090 2024-09-18T00:14:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2091 2024-09-18T00:14:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:47.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:14:47.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:47.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:47.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:14:47.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:14:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:47.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:14:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:47.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:14:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:14:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:14:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:14:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:14:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:14:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:14:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:14:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:14:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:14:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:14:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2092 2024-09-18T00:14:48.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2093 2024-09-18T00:14:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:48.088 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:48.085+0000 7f9bfab49640 -1 reset not still connected to 0x55cd16a2c680 2024-09-18T00:14:48.088 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:48.085+0000 7f9bfab49640 -1 reset not still connected to 0x55cd19c0e680 2024-09-18T00:14:48.089 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:48.086+0000 7f9bfab49640 -1 reset not still connected to 0x55cd22fd5c70 2024-09-18T00:14:48.089 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:48.086+0000 7f9bfab49640 -1 reset not still connected to 0x55cd46da9e10 2024-09-18T00:14:48.090 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:48.088+0000 7f9bfab49640 -1 reset not still connected to 0x55cd4d43a0d0 2024-09-18T00:14:48.091 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:48.089+0000 7f9bfab49640 -1 reset not still connected to 0x55cd504f9450 2024-09-18T00:14:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2097 2024-09-18T00:14:49.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2094 2024-09-18T00:14:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2099 2024-09-18T00:14:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2096 2024-09-18T00:14:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2095 2024-09-18T00:14:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2098 2024-09-18T00:14:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:49.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:14:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:14:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:14:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:14:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:14:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:14:50.062 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:50.059+0000 7f9bfab49640 -1 reset not still connected to 0x55cd22fd5c70 2024-09-18T00:14:50.062 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:50.059+0000 7f9bfab49640 -1 reset not still connected to 0x55cd46da9e10 2024-09-18T00:14:50.062 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:50.059+0000 7f9bfab49640 -1 reset not still connected to 0x55cd4d43a0d0 2024-09-18T00:14:50.062 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:14:50.059+0000 7f9bfab49640 -1 reset not still connected to 0x55cd504f9450 2024-09-18T00:14:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:14:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:14:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:14:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:14:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:14:52.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2100 2024-09-18T00:14:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2101 2024-09-18T00:14:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:14:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:14:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:14:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:14:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:14:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:14:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:14:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:14:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:14:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:14:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:14:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:14:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:14:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:14:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:14:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2103 2024-09-18T00:14:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2102 2024-09-18T00:14:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:53.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2106 2024-09-18T00:14:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2105 2024-09-18T00:14:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2107 2024-09-18T00:14:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2104 2024-09-18T00:14:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2108 2024-09-18T00:14:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2109 2024-09-18T00:14:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:54.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:55.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:14:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:14:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:14:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:14:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:14:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:14:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:14:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:14:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:14:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:14:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:14:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:14:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2111 2024-09-18T00:14:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2110 2024-09-18T00:14:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:14:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:14:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:14:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:14:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:14:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:14:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:14:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:14:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:14:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:14:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:14:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:14:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:14:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:14:57.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:14:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:14:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:14:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:14:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:14:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:14:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:14:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2113 2024-09-18T00:14:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2112 2024-09-18T00:14:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2116 2024-09-18T00:14:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2114 2024-09-18T00:14:59.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2117 2024-09-18T00:14:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2119 2024-09-18T00:14:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2115 2024-09-18T00:14:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2118 2024-09-18T00:14:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:59.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:14:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:15:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:15:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:15:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:15:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:15:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:15:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:15:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:15:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:15:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:15:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:15:01.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:15:02.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2120 2024-09-18T00:15:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2121 2024-09-18T00:15:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:15:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:15:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:15:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:15:02.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:15:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:15:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:15:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:15:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:15:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:15:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:15:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:15:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:15:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:15:02.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:15:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:15:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:15:02.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:15:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:03.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2122 2024-09-18T00:15:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2123 2024-09-18T00:15:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:03.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2124 2024-09-18T00:15:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2127 2024-09-18T00:15:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2128 2024-09-18T00:15:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2126 2024-09-18T00:15:04.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2125 2024-09-18T00:15:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2129 2024-09-18T00:15:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:04.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:04.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:04.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:04.191 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:15:04.189+0000 7f9bfab49640 -1 reset not still connected to 0x55cd15030340 2024-09-18T00:15:04.195 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:15:04.194+0000 7f9bfab49640 -1 reset not still connected to 0x55cd22605380 2024-09-18T00:15:04.195 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:15:04.194+0000 7f9bfab49640 -1 reset not still connected to 0x55cd32ff76c0 2024-09-18T00:15:04.195 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:15:04.194+0000 7f9bfab49640 -1 reset not still connected to 0x55cd34265930 2024-09-18T00:15:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:15:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:15:05.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:15:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:15:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:15:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:15:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:15:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:15:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:15:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:15:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:15:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:15:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2131 2024-09-18T00:15:07.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2130 2024-09-18T00:15:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:15:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:15:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:15:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:15:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:15:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:15:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:15:07.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:15:07.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:15:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:15:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:15:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:15:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:15:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:15:07.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:15:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:15:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:15:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:15:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2133 2024-09-18T00:15:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2132 2024-09-18T00:15:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:08.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2134 2024-09-18T00:15:09.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2136 2024-09-18T00:15:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2139 2024-09-18T00:15:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2138 2024-09-18T00:15:09.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2137 2024-09-18T00:15:09.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2135 2024-09-18T00:15:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:09.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:09.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:15:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:15:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:15:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:15:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:10.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:15:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:15:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:15:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:15:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:15:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:15:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:15:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:15:12.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2140 2024-09-18T00:15:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2141 2024-09-18T00:15:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:15:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:15:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:15:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:15:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:15:12.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:15:12.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:12.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:15:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:15:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:15:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:15:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:15:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:15:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:12.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:15:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:15:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:15:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:15:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:15:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:15:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2143 2024-09-18T00:15:13.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2142 2024-09-18T00:15:13.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:13.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2147 2024-09-18T00:15:14.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2145 2024-09-18T00:15:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2144 2024-09-18T00:15:14.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2148 2024-09-18T00:15:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2146 2024-09-18T00:15:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2149 2024-09-18T00:15:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:14.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:14.230 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:15:14.227+0000 7f9bfab49640 -1 reset not still connected to 0x55cd22fd5c70 2024-09-18T00:15:14.232 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:15:14.228+0000 7f9bfab49640 -1 reset not still connected to 0x55cd46da9e10 2024-09-18T00:15:14.233 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:15:14.228+0000 7f9bfab49640 -1 reset not still connected to 0x55cd4d43a0d0 2024-09-18T00:15:14.233 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:15:14.228+0000 7f9bfab49640 -1 reset not still connected to 0x55cd504f9450 2024-09-18T00:15:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:15:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:15:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:15:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:15.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:15:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:15:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:15:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:15:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:16.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:15:16.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:15:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:15:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:15:16.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:15:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2151 2024-09-18T00:15:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2150 2024-09-18T00:15:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:15:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:15:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:15:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:15:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:15:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:15:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:15:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:15:17.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:15:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:15:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:15:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:15:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:15:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:15:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:15:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:15:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:15:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:15:17.395 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:15:17.394+0000 7f0114255640 -1 reset not still connected to 0x557d0a065e10 2024-09-18T00:15:17.396 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:15:17.394+0000 7f0114255640 -1 reset not still connected to 0x557d0b465110 2024-09-18T00:15:17.397 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:15:17.396+0000 7f0114255640 -1 reset not still connected to 0x557d0b465860 2024-09-18T00:15:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2153 2024-09-18T00:15:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2152 2024-09-18T00:15:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2154 2024-09-18T00:15:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2155 2024-09-18T00:15:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2156 2024-09-18T00:15:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2158 2024-09-18T00:15:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2157 2024-09-18T00:15:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2159 2024-09-18T00:15:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:15:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:15:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:15:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:15:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:15:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:15:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:15:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:15:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:15:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:15:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:15:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:15:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2161 2024-09-18T00:15:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2160 2024-09-18T00:15:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:15:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:15:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:15:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:15:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:15:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:15:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:15:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:15:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:15:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:15:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:15:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:15:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:15:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:15:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:15:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:15:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:15:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:15:23.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2163 2024-09-18T00:15:23.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2162 2024-09-18T00:15:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2167 2024-09-18T00:15:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2165 2024-09-18T00:15:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2166 2024-09-18T00:15:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2168 2024-09-18T00:15:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2164 2024-09-18T00:15:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2169 2024-09-18T00:15:24.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:24.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:15:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:15:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:15:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:15:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:15:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:15:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:15:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:15:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:15:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:15:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:15:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:15:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2171 2024-09-18T00:15:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2170 2024-09-18T00:15:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:15:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:15:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:15:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:15:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:15:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:15:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:15:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:15:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:15:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:15:27.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:15:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:15:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:15:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:15:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:15:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:15:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:15:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:15:28.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2173 2024-09-18T00:15:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2172 2024-09-18T00:15:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:28.191 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:15:28.189+0000 7f9bfab49640 -1 reset not still connected to 0x55cd32ff76c0 2024-09-18T00:15:28.191 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:15:28.189+0000 7f9bfab49640 -1 reset not still connected to 0x55cd34265930 2024-09-18T00:15:29.118 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:15:29.115+0000 7f0114255640 -1 reset not still connected to 0x557d0b464750 2024-09-18T00:15:29.118 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:15:29.115+0000 7f0114255640 -1 reset not still connected to 0x557d0f606c30 2024-09-18T00:15:29.118 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:15:29.115+0000 7f0114255640 -1 reset not still connected to 0x557d1de9b5f0 2024-09-18T00:15:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2177 2024-09-18T00:15:29.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2179 2024-09-18T00:15:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2174 2024-09-18T00:15:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2176 2024-09-18T00:15:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2175 2024-09-18T00:15:29.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2178 2024-09-18T00:15:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:29.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:29.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:15:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:15:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:15:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:15:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:15:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:15:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:15:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:15:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:15:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:15:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:15:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:15:32.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2181 2024-09-18T00:15:32.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2180 2024-09-18T00:15:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:15:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:15:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:15:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:15:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:15:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:15:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:15:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:15:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:15:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:15:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:15:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:15:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:15:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:15:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:15:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:15:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:15:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:15:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2183 2024-09-18T00:15:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2182 2024-09-18T00:15:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2188 2024-09-18T00:15:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2184 2024-09-18T00:15:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2189 2024-09-18T00:15:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2186 2024-09-18T00:15:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2185 2024-09-18T00:15:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2187 2024-09-18T00:15:34.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:34.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:15:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:15:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:15:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:15:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:15:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:15:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:15:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:15:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:15:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:15:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:15:36.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:15:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2190 2024-09-18T00:15:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2191 2024-09-18T00:15:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:15:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:15:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:15:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:15:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:15:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:15:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:15:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:15:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:15:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:15:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:15:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:15:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:15:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:15:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:15:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:15:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:15:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:15:38.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2192 2024-09-18T00:15:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2193 2024-09-18T00:15:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:39.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2194 2024-09-18T00:15:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2199 2024-09-18T00:15:39.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2196 2024-09-18T00:15:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2195 2024-09-18T00:15:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2197 2024-09-18T00:15:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2198 2024-09-18T00:15:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:39.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:39.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:15:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:15:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:15:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:15:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:15:40.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:15:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:15:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:15:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:15:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:15:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:15:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:15:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2200 2024-09-18T00:15:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2201 2024-09-18T00:15:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:15:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:15:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:15:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:15:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:15:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:15:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:15:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:15:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:15:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:15:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:15:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:15:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:15:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:15:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:15:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:15:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:15:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:15:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2202 2024-09-18T00:15:43.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2203 2024-09-18T00:15:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:43.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2206 2024-09-18T00:15:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2207 2024-09-18T00:15:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2209 2024-09-18T00:15:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2205 2024-09-18T00:15:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2204 2024-09-18T00:15:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2208 2024-09-18T00:15:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:15:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:15:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:15:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:15:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:15:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:15:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:15:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:15:46.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:15:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:15:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:15:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:15:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2210 2024-09-18T00:15:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2211 2024-09-18T00:15:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:15:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:15:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:15:47.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:15:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:15:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:15:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:15:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:15:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:15:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:15:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:15:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:15:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:15:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:15:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:15:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:15:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:15:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:15:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2213 2024-09-18T00:15:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2212 2024-09-18T00:15:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2214 2024-09-18T00:15:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2219 2024-09-18T00:15:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2215 2024-09-18T00:15:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2217 2024-09-18T00:15:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2216 2024-09-18T00:15:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2218 2024-09-18T00:15:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:15:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:15:50.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:15:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:15:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:15:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:15:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:15:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:15:51.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:15:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:15:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:15:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:15:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2221 2024-09-18T00:15:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2220 2024-09-18T00:15:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:15:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:15:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:15:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:15:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:15:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:15:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:15:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:15:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:15:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:15:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:15:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:15:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:15:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:15:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:15:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:15:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:15:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:15:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2223 2024-09-18T00:15:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2222 2024-09-18T00:15:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2224 2024-09-18T00:15:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2227 2024-09-18T00:15:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2225 2024-09-18T00:15:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2229 2024-09-18T00:15:54.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2226 2024-09-18T00:15:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2228 2024-09-18T00:15:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:15:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:15:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:15:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:15:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:15:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:15:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:15:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:15:56.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:15:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:15:56.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:15:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2231 2024-09-18T00:15:57.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2230 2024-09-18T00:15:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:15:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:15:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:15:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:15:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:15:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:15:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:15:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:15:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:15:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:15:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:15:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:15:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:15:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:15:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:15:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:15:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:15:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:15:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:15:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:15:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:15:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2233 2024-09-18T00:15:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2232 2024-09-18T00:15:58.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:58.285 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:15:58.283+0000 7f9bfab49640 -1 reset not still connected to 0x55cd34265930 2024-09-18T00:15:58.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2234 2024-09-18T00:15:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2236 2024-09-18T00:15:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2239 2024-09-18T00:15:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2235 2024-09-18T00:15:59.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2238 2024-09-18T00:15:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2237 2024-09-18T00:15:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:59.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:15:59.337 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:15:59.335+0000 7f9bfab49640 -1 reset not still connected to 0x55cd34265930 2024-09-18T00:16:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:16:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:00.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:16:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:16:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:16:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:16:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:16:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:16:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:16:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:16:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:16:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:16:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:16:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2240 2024-09-18T00:16:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2241 2024-09-18T00:16:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:16:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:16:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:16:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:16:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:16:02.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:16:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:16:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:16:02.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:16:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:16:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:16:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:16:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:16:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:16:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:16:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:16:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:16:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:16:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2243 2024-09-18T00:16:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2242 2024-09-18T00:16:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2247 2024-09-18T00:16:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2246 2024-09-18T00:16:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2245 2024-09-18T00:16:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2248 2024-09-18T00:16:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2244 2024-09-18T00:16:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2249 2024-09-18T00:16:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:04.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:16:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:16:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:16:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:16:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:16:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:16:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:16:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:16:06.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:16:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:16:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:16:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:16:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2250 2024-09-18T00:16:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2251 2024-09-18T00:16:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:16:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:16:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:16:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:16:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:16:07.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:16:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:16:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:16:07.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:16:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:16:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:16:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:16:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:16:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:16:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:16:07.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:16:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:16:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:16:08.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2252 2024-09-18T00:16:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2253 2024-09-18T00:16:08.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:08.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2256 2024-09-18T00:16:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2257 2024-09-18T00:16:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2259 2024-09-18T00:16:09.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2258 2024-09-18T00:16:09.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2254 2024-09-18T00:16:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2255 2024-09-18T00:16:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:09.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:16:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:16:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:16:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:16:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:16:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:16:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:16:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:16:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:16:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:16:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:11.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:16:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:16:12.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2260 2024-09-18T00:16:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2261 2024-09-18T00:16:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:16:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:16:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:16:12.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:16:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:16:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:16:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:16:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:16:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:16:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:16:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:16:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:16:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:16:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:16:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:16:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:16:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2262 2024-09-18T00:16:13.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2263 2024-09-18T00:16:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2266 2024-09-18T00:16:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2265 2024-09-18T00:16:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2267 2024-09-18T00:16:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2264 2024-09-18T00:16:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2269 2024-09-18T00:16:14.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2268 2024-09-18T00:16:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:14.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:14.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:16:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:16:15.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:16:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:16:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:16:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:16:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:16:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:16:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:16:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:16.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:16:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:16:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:16:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2271 2024-09-18T00:16:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2270 2024-09-18T00:16:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:16:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:16:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:16:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:16:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:16:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:16:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:16:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:16:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:16:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:16:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:16:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:16:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:16:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:16:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:16:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:16:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:16:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:16:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2273 2024-09-18T00:16:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2272 2024-09-18T00:16:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2275 2024-09-18T00:16:19.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2274 2024-09-18T00:16:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2278 2024-09-18T00:16:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2279 2024-09-18T00:16:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2276 2024-09-18T00:16:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2277 2024-09-18T00:16:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:19.437 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:16:19.432+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0adfb930 2024-09-18T00:16:19.437 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:16:19.432+0000 7f9bfab49640 -1 reset not still connected to 0x55cd15030340 2024-09-18T00:16:19.438 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:16:19.435+0000 7f9bfab49640 -1 reset not still connected to 0x55cd22605380 2024-09-18T00:16:19.438 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:16:19.435+0000 7f9bfab49640 -1 reset not still connected to 0x55cd32ff76c0 2024-09-18T00:16:19.438 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:16:19.435+0000 7f9bfab49640 -1 reset not still connected to 0x55cd34265930 2024-09-18T00:16:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:16:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:16:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:16:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:16:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:16:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:16:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:16:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:16:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:16:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:16:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:16:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:16:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2280 2024-09-18T00:16:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2281 2024-09-18T00:16:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:16:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:16:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:16:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:16:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:16:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:16:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:16:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:16:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:16:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:16:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:16:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:16:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:16:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:16:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:16:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:16:22.464 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:16:22.461+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0b7f6000 2024-09-18T00:16:22.464 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:16:22.461+0000 7f9bfab49640 -1 reset not still connected to 0x55cd15030340 2024-09-18T00:16:22.465 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:16:22.462+0000 7f9bfab49640 -1 reset not still connected to 0x55cd22605380 2024-09-18T00:16:22.465 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:16:22.462+0000 7f9bfab49640 -1 reset not still connected to 0x55cd32ff76c0 2024-09-18T00:16:22.465 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:16:22.462+0000 7f9bfab49640 -1 reset not still connected to 0x55cd34265930 2024-09-18T00:16:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2282 2024-09-18T00:16:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2283 2024-09-18T00:16:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2286 2024-09-18T00:16:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2287 2024-09-18T00:16:24.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2285 2024-09-18T00:16:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2289 2024-09-18T00:16:24.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2284 2024-09-18T00:16:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2288 2024-09-18T00:16:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:16:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:16:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:16:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:16:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 106892 2024-09-18T00:16:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:16:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:16:26.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:16:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:16:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 107024 2024-09-18T00:16:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2290 2024-09-18T00:16:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:16:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:16:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:16:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:16:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:16:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:16:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:16:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 107156 2024-09-18T00:16:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:16:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 107308 2024-09-18T00:16:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:16:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 107460 2024-09-18T00:16:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:16:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:16:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:16:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:16:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:16:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:16:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2291 2024-09-18T00:16:28.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2294 2024-09-18T00:16:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2293 2024-09-18T00:16:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2292 2024-09-18T00:16:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:16:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:16:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 107613 2024-09-18T00:16:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:16:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 107766 2024-09-18T00:16:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:16:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 107919 2024-09-18T00:16:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:16:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 108092 2024-09-18T00:16:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:16:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:16:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 106761 2024-09-18T00:18:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:18:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:18:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:18:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 106893 2024-09-18T00:18:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:18:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:18:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:18:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 107025 2024-09-18T00:18:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:18:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:18:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:18:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 107157 2024-09-18T00:18:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:18:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:18:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:18:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 107309 2024-09-18T00:19:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:19:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:19:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:19:00.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 107461 2024-09-18T00:19:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:19:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:19:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:19:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 107614 2024-09-18T00:19:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:19:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:19:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:19:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 107767 2024-09-18T00:19:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:19:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:19:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:19:03.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 107920 2024-09-18T00:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:19:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 108093 2024-09-18T00:19:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:19:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:19:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:19:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:19:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:19:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:19:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 1 2024-09-18T00:19:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd0 2024-09-18T00:19:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary1 2024-09-18T00:19:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary1 2024-09-18T00:19:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:19:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:19:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary1 2024-09-18T00:19:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:19:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 2 2024-09-18T00:19:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd1 2024-09-18T00:19:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary2 2024-09-18T00:19:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary2 2024-09-18T00:19:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary2 2024-09-18T00:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 3 2024-09-18T00:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd2 2024-09-18T00:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary3 2024-09-18T00:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary3 2024-09-18T00:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary3 2024-09-18T00:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:19:05.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 4 2024-09-18T00:19:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd3 2024-09-18T00:19:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary4 2024-09-18T00:19:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary4 2024-09-18T00:19:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:19:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:19:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary4 2024-09-18T00:19:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:19:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 5 2024-09-18T00:19:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:19:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd4 2024-09-18T00:19:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary5 2024-09-18T00:19:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:19:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary5 2024-09-18T00:19:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:19:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary5 2024-09-18T00:19:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 6 2024-09-18T00:19:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd5 2024-09-18T00:19:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:19:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:19:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 7 2024-09-18T00:19:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary6 2024-09-18T00:19:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary6 2024-09-18T00:19:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:19:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary6 2024-09-18T00:19:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd6 2024-09-18T00:19:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary7 2024-09-18T00:19:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary7 2024-09-18T00:19:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:19:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:19:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary7 2024-09-18T00:19:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:19:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:19:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:19:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:19:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:19:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 114816 2024-09-18T00:19:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 10 2024-09-18T00:19:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd9 2024-09-18T00:19:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary10 2024-09-18T00:19:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary10 2024-09-18T00:19:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:19:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary10 2024-09-18T00:19:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 9 2024-09-18T00:19:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd8 2024-09-18T00:19:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary9 2024-09-18T00:19:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary9 2024-09-18T00:19:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:19:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary9 2024-09-18T00:19:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 8 2024-09-18T00:19:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd7 2024-09-18T00:19:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary8 2024-09-18T00:19:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary8 2024-09-18T00:19:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:19:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary8 2024-09-18T00:19:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-18T00:19:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-18T00:19:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-18T00:19:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-18T00:19:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-18T00:19:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-18T00:19:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-18T00:19:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-18T00:19:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-18T00:19:05.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-18T00:19:05.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=33a665e420d24544204423a7673461c6 2024-09-18T00:19:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-18T00:19:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-18T00:19:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:19:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-18T00:19:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=a24cc64ea908616d9ed8a41ec640a22e 2024-09-18T00:19:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-18T00:19:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-18T00:19:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:19:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-18T00:19:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=afde990fb8820a1f96119755d4ce25b9 2024-09-18T00:19:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-18T00:19:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-18T00:19:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:19:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-18T00:19:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=f275e62d7bdf5a926dd1307560180f6f 2024-09-18T00:19:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-18T00:19:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-18T00:19:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:19:36.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-18T00:19:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=3e77880a1e8059c4fd65a3127de95f4a 2024-09-18T00:19:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-18T00:19:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:19:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:19:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-18T00:19:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:19:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-18T00:19:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:19:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-18T00:19:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:19:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-18T00:19:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=b716960e1116b1e369f95ae466b5ea55 2024-09-18T00:19:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-18T00:19:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-18T00:19:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:19:37.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-18T00:19:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=cddf1c0be835ed196e3cd01aa3832877 2024-09-18T00:19:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-18T00:19:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=6c111329c32ead50f508cf8394763b96 2024-09-18T00:19:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-18T00:19:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=a8444237d6fe0ea0d23997c8d2528182 2024-09-18T00:19:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-18T00:19:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=154327eb4d92fb36c16c03376ae74e47 2024-09-18T00:19:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-18T00:19:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-18T00:19:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:19:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-18T00:19:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-18T00:19:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:19:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-18T00:19:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-18T00:19:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:19:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-18T00:19:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-18T00:19:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:19:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-18T00:19:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:19:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:19:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-18T00:19:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:19:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-18T00:19:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:19:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-18T00:19:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-18T00:19:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:19:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:19:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-18T00:19:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:19:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:19:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d2b68ce6-80c4-4597-8006-4885dadbe8d9 2024-09-18T00:19:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37' 2024-09-18T00:19:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:37' 2024-09-18T00:19:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-18 00:19:37' 2024-09-18T00:19:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:19:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:37' 2024-09-18T00:19:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:19:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 98347238-de9a-416e-80a1-1bb76d54ed82 2024-09-18T00:19:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37' 2024-09-18T00:19:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:37' 2024-09-18T00:19:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-18 00:19:37' 2024-09-18T00:19:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:19:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:37' 2024-09-18T00:19:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING4642634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":137216.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618581,"remote_snapshot_timestamp":1726618581,"replay_state":"idle"}2024-09-18 00:19:17image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:37image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3516416.0,"last_snapshot_bytes":5341184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618582,"remote_snapshot_timestamp":1726618582,"replay_state":"idle"}2024-09-18 00:19:17image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":137216.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618583,"remote_snapshot_timestamp":1726618583,"replay_state":"idle"}2024-09-18 00:19:17image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:37image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:19:37image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3661824.0,"last_snapshot_bytes":126976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618586,"remote_snapshot_timestamp":1726618586,"replay_state":"idle"}2024-09-18 00:19:17image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3395584.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618587,"remote_snapshot_timestamp":1726618587,"replay_state":"idle"}2024-09-18 00:19:17image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3377152.0,"last_snapshot_bytes":5197824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618588,"remote_snapshot_timestamp":1726618588,"replay_state":"idle"}2024-09-18 00:19:17image-primary9814425c6-1326-454a-85d2-08f9f100d357up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2658304.0,"last_snapshot_bytes":5181440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618588,"remote_snapshot_timestamp":1726618588,"replay_state":"idle"}2024-09-18 00:19:17' 2024-09-18T00:19:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:19:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:37' 2024-09-18T00:19:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:19:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-18T00:19:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:19:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING4642634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":137216.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618581,"remote_snapshot_timestamp":1726618581,"replay_state":"idle"}2024-09-18 00:19:17image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:37image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3516416.0,"last_snapshot_bytes":5341184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618582,"remote_snapshot_timestamp":1726618582,"replay_state":"idle"}2024-09-18 00:19:17image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":137216.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618583,"remote_snapshot_timestamp":1726618583,"replay_state":"idle"}2024-09-18 00:19:17image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:37image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:19:37image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3661824.0,"last_snapshot_bytes":126976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618586,"remote_snapshot_timestamp":1726618586,"replay_state":"idle"}2024-09-18 00:19:17image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3395584.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618587,"remote_snapshot_timestamp":1726618587,"replay_state":"idle"}2024-09-18 00:19:17image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3377152.0,"last_snapshot_bytes":5197824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618588,"remote_snapshot_timestamp":1726618588,"replay_state":"idle"}2024-09-18 00:19:17image-primary9814425c6-1326-454a-85d2-08f9f100d357up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2658304.0,"last_snapshot_bytes":5181440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618588,"remote_snapshot_timestamp":1726618588,"replay_state":"idle"}2024-09-18 00:19:17' 2024-09-18T00:19:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:38.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:19:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:37' 2024-09-18T00:19:38.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:19:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-18T00:19:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:19:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:19:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:19:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:19:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:19:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:19:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:19:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-18T00:19:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:19:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:19:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-18T00:19:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:19:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-18T00:19:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-18T00:19:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:19:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-18T00:19:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:19:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-18T00:19:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:19:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-18T00:19:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:19:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-18T00:19:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:19:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:38.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-18T00:19:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:19:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:19:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: eff149be-58c6-4f92-96d5-04d6dc49412e 2024-09-18T00:19:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:19:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37' 2024-09-18T00:19:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:39.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:39.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:39.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:37' 2024-09-18T00:19:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-18 00:19:37' 2024-09-18T00:19:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:19:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:37' 2024-09-18T00:19:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:19:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a827873c-19b1-443f-b747-bed14c3f54c3 2024-09-18T00:19:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:19:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:39' 2024-09-18T00:19:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:39.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:37' 2024-09-18T00:19:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-18 00:19:37' 2024-09-18T00:19:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:39.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:19:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:37' 2024-09-18T00:19:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:39.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-18T00:19:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:19:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-18T00:19:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-18T00:19:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:19:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-18T00:19:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING4642634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":137216.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618581,"remote_snapshot_timestamp":1726618581,"replay_state":"idle"}2024-09-18 00:19:17image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:37image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3516416.0,"last_snapshot_bytes":5341184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618582,"remote_snapshot_timestamp":1726618582,"replay_state":"idle"}2024-09-18 00:19:17image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":137216.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618583,"remote_snapshot_timestamp":1726618583,"replay_state":"idle"}2024-09-18 00:19:17image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:37image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:19:37image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3661824.0,"last_snapshot_bytes":126976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618586,"remote_snapshot_timestamp":1726618586,"replay_state":"idle"}2024-09-18 00:19:17image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3395584.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618587,"remote_snapshot_timestamp":1726618587,"replay_state":"idle"}2024-09-18 00:19:17image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:19:39image-primary9814425c6-1326-454a-85d2-08f9f100d357up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2658304.0,"last_snapshot_bytes":5181440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618588,"remote_snapshot_timestamp":1726618588,"replay_state":"idle"}2024-09-18 00:19:17' 2024-09-18T00:19:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:19:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:37' 2024-09-18T00:19:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:19:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:39.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-18T00:19:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:19:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING4642634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":137216.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618581,"remote_snapshot_timestamp":1726618581,"replay_state":"idle"}2024-09-18 00:19:17image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:37image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3516416.0,"last_snapshot_bytes":5341184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618582,"remote_snapshot_timestamp":1726618582,"replay_state":"idle"}2024-09-18 00:19:17image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":137216.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618583,"remote_snapshot_timestamp":1726618583,"replay_state":"idle"}2024-09-18 00:19:17image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:37image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:19:37image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3661824.0,"last_snapshot_bytes":126976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618586,"remote_snapshot_timestamp":1726618586,"replay_state":"idle"}2024-09-18 00:19:17image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3395584.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618587,"remote_snapshot_timestamp":1726618587,"replay_state":"idle"}2024-09-18 00:19:17image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:19:39image-primary9814425c6-1326-454a-85d2-08f9f100d357up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:19:37cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2658304.0,"last_snapshot_bytes":5181440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618588,"remote_snapshot_timestamp":1726618588,"replay_state":"idle"}2024-09-18 00:19:17' 2024-09-18T00:19:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:19:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:37' 2024-09-18T00:19:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:19:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:39.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-18T00:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:39.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:19:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:19:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 98347238-de9a-416e-80a1-1bb76d54ed82 2024-09-18T00:19:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:19:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37' 2024-09-18T00:19:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:39.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:39.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:37' 2024-09-18T00:19:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-18 00:19:37' 2024-09-18T00:19:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:19:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:37' 2024-09-18T00:19:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:39.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:19:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d2b68ce6-80c4-4597-8006-4885dadbe8d9 2024-09-18T00:19:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:19:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37' 2024-09-18T00:19:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:37' 2024-09-18T00:19:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-18 00:19:37' 2024-09-18T00:19:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:19:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:37' 2024-09-18T00:19:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-18T00:19:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:19:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:39.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:19:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-18T00:19:39.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:19:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-18T00:19:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:39.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:19:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-18T00:19:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-18T00:19:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:19:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:19:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:19:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:19:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-18T00:19:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:19:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING4642604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":137216.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618581,"remote_snapshot_timestamp":1726618581,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3516416.0,"last_snapshot_bytes":5341184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618582,"remote_snapshot_timestamp":1726618582,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":137216.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618583,"remote_snapshot_timestamp":1726618583,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:19:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3661824.0,"last_snapshot_bytes":126976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618586,"remote_snapshot_timestamp":1726618586,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3395584.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618587,"remote_snapshot_timestamp":1726618587,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:19:39cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary9814425c6-1326-454a-85d2-08f9f100d357up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2658304.0,"last_snapshot_bytes":5181440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618588,"remote_snapshot_timestamp":1726618588,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37' 2024-09-18T00:19:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:19:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:19:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:19:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:19:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:37' 2024-09-18T00:19:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:19:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-18T00:19:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:19:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:19:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-18T00:19:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:19:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 814425c6-1326-454a-85d2-08f9f100d357 2024-09-18T00:19:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:19:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:19:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:19:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2658304.0,"last_snapshot_bytes":5181440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618588,"remote_snapshot_timestamp":1726618588,"replay_state":"idle"} 2024-09-18T00:19:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:17' 2024-09-18T00:19:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:19:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:40.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:19:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:19:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 02b583ab-f4b1-4c02-b3ba-6dd65cd3316c 2024-09-18T00:19:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:19:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:19:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopping_replay 2024-09-18T00:19:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:19:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:40' 2024-09-18T00:19:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:19:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:19:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:19:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 5aa63b87-d45e-4d8e-a667-5dafb6b4ef7c 2024-09-18T00:19:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:19:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:19:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:19:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":137216.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618583,"remote_snapshot_timestamp":1726618583,"replay_state":"idle"} 2024-09-18T00:19:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:17' 2024-09-18T00:19:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:40.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:19:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:19:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:19:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6 2024-09-18T00:19:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:19:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:19:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:19:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3661824.0,"last_snapshot_bytes":126976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618586,"remote_snapshot_timestamp":1726618586,"replay_state":"idle"} 2024-09-18T00:19:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:17' 2024-09-18T00:19:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:19:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4 2024-09-18T00:19:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:19:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:19:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopping_replay 2024-09-18T00:19:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:19:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:40' 2024-09-18T00:19:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:19:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:40.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:19:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:19:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:19:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:19:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d64cf4b9-98c4-4f70-a161-76e8c07ce938 2024-09-18T00:19:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:19:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:19:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:19:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:40' 2024-09-18T00:19:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:19:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:19:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING4642604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":137216.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618581,"remote_snapshot_timestamp":1726618581,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3516416.0,"last_snapshot_bytes":5341184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618582,"remote_snapshot_timestamp":1726618582,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":137216.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618583,"remote_snapshot_timestamp":1726618583,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:19:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3661824.0,"last_snapshot_bytes":126976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618586,"remote_snapshot_timestamp":1726618586,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:19:40cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:19:39cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary9814425c6-1326-454a-85d2-08f9f100d357up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2658304.0,"last_snapshot_bytes":5181440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618588,"remote_snapshot_timestamp":1726618588,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37' 2024-09-18T00:19:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:40.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:19:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:37' 2024-09-18T00:19:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:19:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-18T00:19:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:19:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:19:40.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-18T00:19:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-18T00:19:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:19:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-18T00:19:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:19:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: eff149be-58c6-4f92-96d5-04d6dc49412e 2024-09-18T00:19:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:19:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:19:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37' 2024-09-18T00:19:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-18T00:19:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:19:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-18T00:19:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:19:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:37' 2024-09-18T00:19:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-18 00:19:37' 2024-09-18T00:19:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:19:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:37' 2024-09-18T00:19:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:40.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:19:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a827873c-19b1-443f-b747-bed14c3f54c3 2024-09-18T00:19:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:19:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:39 2024-09-18T00:19:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:19:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37' 2024-09-18T00:19:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:19:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:40.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:40.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:40.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:39' 2024-09-18T00:19:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-18 00:19:39' 2024-09-18T00:19:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:19:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:39' 2024-09-18T00:19:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING7342604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:19:40cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:19:40cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":137216.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618583,"remote_snapshot_timestamp":1726618583,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:19:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3661824.0,"last_snapshot_bytes":126976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618586,"remote_snapshot_timestamp":1726618586,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:19:40cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:19:39cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary9814425c6-1326-454a-85d2-08f9f100d357up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2658304.0,"last_snapshot_bytes":5181440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618588,"remote_snapshot_timestamp":1726618588,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37' 2024-09-18T00:19:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:19:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:37' 2024-09-18T00:19:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:19:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:40.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-18T00:19:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:19:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:19:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-18T00:19:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING7342604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:19:40cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:19:40cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":137216.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618583,"remote_snapshot_timestamp":1726618583,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:19:37cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3661824.0,"last_snapshot_bytes":126976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618586,"remote_snapshot_timestamp":1726618586,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:19:40cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:19:39cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:37image-primary9814425c6-1326-454a-85d2-08f9f100d357up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2658304.0,"last_snapshot_bytes":5181440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618588,"remote_snapshot_timestamp":1726618588,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:19:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:19:37' 2024-09-18T00:19:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:19:40.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:39' 2024-09-18T00:19:40.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:19:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-18T00:19:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:19:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:19:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-18T00:19:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:19:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:19:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:19:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:19:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:19:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:19:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-18T00:19:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:19:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:19:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-18T00:19:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:19:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:19:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-18T00:19:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:19:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:19:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-18T00:19:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd0 2024-09-18T00:19:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-18T00:19:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd1 2024-09-18T00:19:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-18T00:19:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd2 2024-09-18T00:19:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-18T00:19:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:41.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd3 2024-09-18T00:19:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-18T00:19:41.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:42.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-18T00:19:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:19:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:19:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-18T00:19:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:19:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:42.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:19:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-18T00:19:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:19:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:42.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:19:42.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-18T00:19:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:19:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:42.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:19:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-18T00:19:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:19:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:42.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:19:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-18T00:19:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:19:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:42.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:19:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:19:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 814425c6-1326-454a-85d2-08f9f100d357 2024-09-18T00:19:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:19:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:19:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:19:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:41' 2024-09-18T00:19:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:19:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:19:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 02b583ab-f4b1-4c02-b3ba-6dd65cd3316c 2024-09-18T00:19:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:19:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:19:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:19:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:40' 2024-09-18T00:19:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 4 2024-09-18T00:19:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:19:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 4 2024-09-18T00:19:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:19:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 5aa63b87-d45e-4d8e-a667-5dafb6b4ef7c 2024-09-18T00:19:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:19:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:19:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:19:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:40' 2024-09-18T00:19:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:19:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 4 2024-09-18T00:19:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:19:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6 2024-09-18T00:19:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:19:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:19:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:19:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:41' 2024-09-18T00:19:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:19:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 4 2024-09-18T00:19:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:19:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4 2024-09-18T00:19:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:19:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:19:42.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:19:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:40' 2024-09-18T00:19:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:19:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 4 2024-09-18T00:19:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:19:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d64cf4b9-98c4-4f70-a161-76e8c07ce938 2024-09-18T00:19:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:19:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:19:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:37 2024-09-18T00:19:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:19:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:40' 2024-09-18T00:19:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:19:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 4 2024-09-18T00:19:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-18T00:19:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:19:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:19:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-18T00:19:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:19:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:19:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-18T00:19:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:19:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:19:46.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-18T00:19:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:19:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:19:46.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-18T00:19:46.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:46.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:19:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-18T00:19:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:19:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:19:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:19:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:19:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 814425c6-1326-454a-85d2-08f9f100d357 2024-09-18T00:19:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:42 2024-09-18T00:19:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:42' 2024-09-18T00:19:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:46.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:19:46.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 5aa63b87-d45e-4d8e-a667-5dafb6b4ef7c 2024-09-18T00:19:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:42 2024-09-18T00:19:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:42' 2024-09-18T00:19:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:19:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 02b583ab-f4b1-4c02-b3ba-6dd65cd3316c 2024-09-18T00:19:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:42 2024-09-18T00:19:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:42' 2024-09-18T00:19:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:46.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:46.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:46.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:46.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:46.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:42' 2024-09-18T00:19:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-18 00:19:42' 2024-09-18T00:19:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:19:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:42' 2024-09-18T00:19:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:42' 2024-09-18T00:19:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-18 00:19:42' 2024-09-18T00:19:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:19:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:42' 2024-09-18T00:19:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:42' 2024-09-18T00:19:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-18 00:19:42' 2024-09-18T00:19:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:19:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:42' 2024-09-18T00:19:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:19:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6 2024-09-18T00:19:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:42 2024-09-18T00:19:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:42' 2024-09-18T00:19:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:19:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4 2024-09-18T00:19:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:42 2024-09-18T00:19:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:42' 2024-09-18T00:19:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:19:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d64cf4b9-98c4-4f70-a161-76e8c07ce938 2024-09-18T00:19:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:42 2024-09-18T00:19:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:19:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:42' 2024-09-18T00:19:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:46.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:46.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:42' 2024-09-18T00:19:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-18 00:19:42' 2024-09-18T00:19:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:19:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:42' 2024-09-18T00:19:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:46.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:46.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:46.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:42' 2024-09-18T00:19:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-18 00:19:42' 2024-09-18T00:19:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:19:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:42' 2024-09-18T00:19:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:42' 2024-09-18T00:19:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-18 00:19:42' 2024-09-18T00:19:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:19:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:19:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:42' 2024-09-18T00:19:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:46.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING6442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary9814425c6-1326-454a-85d2-08f9f100d357up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42' 2024-09-18T00:19:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:19:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:42' 2024-09-18T00:19:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:19:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:46.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:46.516 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-18T00:19:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:19:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING6442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary9814425c6-1326-454a-85d2-08f9f100d357up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42' 2024-09-18T00:19:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:19:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:42' 2024-09-18T00:19:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:19:46.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:46.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-18T00:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING6442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary9814425c6-1326-454a-85d2-08f9f100d357up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42' 2024-09-18T00:19:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:19:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:42' 2024-09-18T00:19:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:19:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-18T00:19:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:19:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING6442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary9814425c6-1326-454a-85d2-08f9f100d357up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42' 2024-09-18T00:19:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:46.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:19:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING6442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary9814425c6-1326-454a-85d2-08f9f100d357up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42' 2024-09-18T00:19:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:19:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:42' 2024-09-18T00:19:46.602 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:19:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:42' 2024-09-18T00:19:46.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:19:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-18T00:19:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:19:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-18T00:19:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:19:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING6442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:19:42image-primary9814425c6-1326-454a-85d2-08f9f100d357up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:19:42cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:19:42' 2024-09-18T00:19:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:19:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:42' 2024-09-18T00:19:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:19:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-18T00:19:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:19:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:19:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:19:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-18T00:19:47.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:19:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-18T00:19:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-18T00:19:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:19:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-18T00:19:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-18T00:19:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:19:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-18T00:19:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:19:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 5aa63b87-d45e-4d8e-a667-5dafb6b4ef7c 2024-09-18T00:19:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:47 2024-09-18T00:19:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:19:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:47' 2024-09-18T00:19:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:47.606 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-18T00:19:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:19:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-18T00:19:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:47.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:19:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-18T00:19:47.615 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-18T00:19:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:47.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:47' 2024-09-18T00:19:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-18 00:19:47' 2024-09-18T00:19:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:19:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:47' 2024-09-18T00:19:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-18T00:19:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:19:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:19:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:19:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-18T00:19:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:19:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 02b583ab-f4b1-4c02-b3ba-6dd65cd3316c 2024-09-18T00:19:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:47 2024-09-18T00:19:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:19:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:47' 2024-09-18T00:19:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:19:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 814425c6-1326-454a-85d2-08f9f100d357 2024-09-18T00:19:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:47 2024-09-18T00:19:47.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:19:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:47' 2024-09-18T00:19:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:47.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:47.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:47.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:47.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:47' 2024-09-18T00:19:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-18 00:19:47' 2024-09-18T00:19:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:19:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:47' 2024-09-18T00:19:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:47' 2024-09-18T00:19:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-18 00:19:47' 2024-09-18T00:19:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:19:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:47' 2024-09-18T00:19:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:19:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6 2024-09-18T00:19:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:47 2024-09-18T00:19:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:19:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:47' 2024-09-18T00:19:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:19:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d64cf4b9-98c4-4f70-a161-76e8c07ce938 2024-09-18T00:19:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:47 2024-09-18T00:19:47.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:19:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:47' 2024-09-18T00:19:47.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:47.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:19:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4 2024-09-18T00:19:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:19:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:47 2024-09-18T00:19:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:19:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:19:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:19:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:19:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:19:47' 2024-09-18T00:19:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:19:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:19:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:19:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:47' 2024-09-18T00:19:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-18 00:19:47' 2024-09-18T00:19:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:19:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:19:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:19:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:19:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:47' 2024-09-18T00:19:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:19:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:47' 2024-09-18T00:19:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-18 00:19:47' 2024-09-18T00:19:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:19:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:47' 2024-09-18T00:19:47.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:19:47.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:19:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:19:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:47' 2024-09-18T00:19:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-18 00:19:47' 2024-09-18T00:19:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:19:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:19:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:19:47' 2024-09-18T00:19:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:19:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:19:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:19:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING6442604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":8302592.0,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":11898880.0,"bytes_per_snapshot":61140992.0,"last_snapshot_bytes":61140992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":28991488.0,"bytes_per_snapshot":66154496.0,"last_snapshot_bytes":66154496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":13266944.0,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary9814425c6-1326-454a-85d2-08f9f100d357up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47' 2024-09-18T00:19:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:19:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:47' 2024-09-18T00:19:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:19:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-18T00:19:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:19:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:19:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-18T00:19:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING6442604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":8302592.0,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":11898880.0,"bytes_per_snapshot":61140992.0,"last_snapshot_bytes":61140992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":28991488.0,"bytes_per_snapshot":66154496.0,"last_snapshot_bytes":66154496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":13266944.0,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary9814425c6-1326-454a-85d2-08f9f100d357up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47' 2024-09-18T00:19:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING6442604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":8302592.0,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":11898880.0,"bytes_per_snapshot":61140992.0,"last_snapshot_bytes":61140992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":28991488.0,"bytes_per_snapshot":66154496.0,"last_snapshot_bytes":66154496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":13266944.0,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary9814425c6-1326-454a-85d2-08f9f100d357up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47' 2024-09-18T00:19:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:47.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:19:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:19:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:47' 2024-09-18T00:19:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:19:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:47' 2024-09-18T00:19:48.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:48.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:48.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:48.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:48.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-18T00:19:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:19:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:19:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-18T00:19:48.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:19:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-18T00:19:48.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:19:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:19:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-18T00:19:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING6442604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":8302592.0,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":11898880.0,"bytes_per_snapshot":61140992.0,"last_snapshot_bytes":61140992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":28991488.0,"bytes_per_snapshot":66154496.0,"last_snapshot_bytes":66154496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":13266944.0,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary9814425c6-1326-454a-85d2-08f9f100d357up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47' 2024-09-18T00:19:48.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:19:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:47' 2024-09-18T00:19:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:19:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:48.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-18T00:19:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:19:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:19:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-18T00:19:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING6442604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":8302592.0,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":11898880.0,"bytes_per_snapshot":61140992.0,"last_snapshot_bytes":61140992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":28991488.0,"bytes_per_snapshot":66154496.0,"last_snapshot_bytes":66154496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":13266944.0,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary9814425c6-1326-454a-85d2-08f9f100d357up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47' 2024-09-18T00:19:48.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:19:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:47' 2024-09-18T00:19:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:19:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING6442604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":8302592.0,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":11898880.0,"bytes_per_snapshot":61140992.0,"last_snapshot_bytes":61140992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":28991488.0,"bytes_per_snapshot":66154496.0,"last_snapshot_bytes":66154496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":13266944.0,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}2024-09-18 00:19:47image-primary9814425c6-1326-454a-85d2-08f9f100d357up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:19:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:19:47' 2024-09-18T00:19:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:19:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:19:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:48.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:19:47' 2024-09-18T00:19:48.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-18T00:19:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:19:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:19:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:19:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-18T00:19:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:19:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:19:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:19:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:19:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:19:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-18T00:19:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:19:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:19:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:19:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:19:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-18T00:19:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:19:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:19:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:19:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:19:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:19:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:19:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:19:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:19:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:19:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:19:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-18T00:19:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-18T00:19:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:19:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:19:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-18T00:19:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:19:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:19:48.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-18T00:19:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:19:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:19:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:19:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:19:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-18T00:19:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-18T00:19:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd4 2024-09-18T00:19:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-18T00:19:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd8 2024-09-18T00:19:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-18T00:19:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd5 2024-09-18T00:19:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-18T00:19:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd6 2024-09-18T00:19:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd7 2024-09-18T00:19:49.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-18T00:19:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-18T00:19:49.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd9 2024-09-18T00:19:49.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:19:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-18T00:20:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=f275e62d7bdf5a926dd1307560180f6f 2024-09-18T00:20:08.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-18T00:20:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=a24cc64ea908616d9ed8a41ec640a22e 2024-09-18T00:20:08.599 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-18T00:20:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ f275e62d7bdf5a926dd1307560180f6f != \f\2\7\5\e\6\2\d\7\b\d\f\5\a\9\2\6\d\d\1\3\0\7\5\6\0\1\8\0\f\6\f ]] 2024-09-18T00:20:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ a24cc64ea908616d9ed8a41ec640a22e != \a\2\4\c\c\6\4\e\a\9\0\8\6\1\6\d\9\e\d\8\a\4\1\e\c\6\4\0\a\2\2\e ]] 2024-09-18T00:20:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=33a665e420d24544204423a7673461c6 2024-09-18T00:20:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-18T00:20:08.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 33a665e420d24544204423a7673461c6 != \3\3\a\6\6\5\e\4\2\0\d\2\4\5\4\4\2\0\4\4\2\3\a\7\6\7\3\4\6\1\c\6 ]] 2024-09-18T00:20:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=afde990fb8820a1f96119755d4ce25b9 2024-09-18T00:20:09.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-18T00:20:09.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ afde990fb8820a1f96119755d4ce25b9 != \a\f\d\e\9\9\0\f\b\8\8\2\0\a\1\f\9\6\1\1\9\7\5\5\d\4\c\e\2\5\b\9 ]] 2024-09-18T00:20:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=cddf1c0be835ed196e3cd01aa3832877 2024-09-18T00:20:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-18T00:20:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ cddf1c0be835ed196e3cd01aa3832877 != \c\d\d\f\1\c\0\b\e\8\3\5\e\d\1\9\6\e\3\c\d\0\1\a\a\3\8\3\2\8\7\7 ]] 2024-09-18T00:20:17.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=a8444237d6fe0ea0d23997c8d2528182 2024-09-18T00:20:17.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-18T00:20:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ a8444237d6fe0ea0d23997c8d2528182 != \a\8\4\4\4\2\3\7\d\6\f\e\0\e\a\0\d\2\3\9\9\7\c\8\d\2\5\2\8\1\8\2 ]] 2024-09-18T00:20:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=3e77880a1e8059c4fd65a3127de95f4a 2024-09-18T00:20:17.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-18T00:20:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=b716960e1116b1e369f95ae466b5ea55 2024-09-18T00:20:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-18T00:20:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 3e77880a1e8059c4fd65a3127de95f4a != \3\e\7\7\8\8\0\a\1\e\8\0\5\9\c\4\f\d\6\5\a\3\1\2\7\d\e\9\5\f\4\a ]] 2024-09-18T00:20:17.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ b716960e1116b1e369f95ae466b5ea55 != \b\7\1\6\9\6\0\e\1\1\1\6\b\1\e\3\6\9\f\9\5\a\e\4\6\6\b\5\e\a\5\5 ]] 2024-09-18T00:20:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=6c111329c32ead50f508cf8394763b96 2024-09-18T00:20:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-18T00:20:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 6c111329c32ead50f508cf8394763b96 != \6\c\1\1\1\3\2\9\c\3\2\e\a\d\5\0\f\5\0\8\c\f\8\3\9\4\7\6\3\b\9\6 ]] 2024-09-18T00:20:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:20:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 114817 2024-09-18T00:20:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:20:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 114819 2024-09-18T00:20:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:20:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 114821 2024-09-18T00:20:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:20:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 114823 2024-09-18T00:20:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:20:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 114825 2024-09-18T00:20:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:20:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 114826 2024-09-18T00:20:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=154327eb4d92fb36c16c03376ae74e47 2024-09-18T00:20:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-18T00:20:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 154327eb4d92fb36c16c03376ae74e47 != \1\5\4\3\2\7\e\b\4\d\9\2\f\b\3\6\c\1\6\c\0\3\3\7\6\a\e\7\4\e\4\7 ]] 2024-09-18T00:20:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:20:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 114829 2024-09-18T00:20:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:20:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 114830 2024-09-18T00:20:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:20:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 114832 2024-09-18T00:20:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-18T00:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-18T00:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-18T00:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:20:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-18T00:20:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:20:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:20:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:20:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4 2024-09-18T00:20:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:20:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3335168.0,"bytes_per_snapshot":66703360.0,"last_snapshot_bytes":66703360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"} 2024-09-18T00:20:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:20:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:17 2024-09-18T00:20:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:20:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:20:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:20:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:20:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:17' 2024-09-18T00:20:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:20:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:20:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:20:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3335168.0,"bytes_per_snapshot":66703360.0,"last_snapshot_bytes":66703360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"} 2024-09-18T00:20:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:20:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:20:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:20:19.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:20:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:20:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:20:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-18 00:20:17' 2024-09-18T00:20:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:20:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:20:17' 2024-09-18T00:20:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:20:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary19c8a07e4-93b7-4a6f-aa6c-17a2e6a50ce4up+replayingreplaying, {"bytes_per_second":3335168.0,"bytes_per_snapshot":66703360.0,"last_snapshot_bytes":66703360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":2106845.87,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+replayingreplaying, {"bytes_per_second":3159859.2,"bytes_per_snapshot":63197184.0,"last_snapshot_bytes":63197184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+replayingreplaying, {"bytes_per_second":3337625.6,"bytes_per_snapshot":66752512.0,"last_snapshot_bytes":66752512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+replayingreplaying, {"bytes_per_second":2038033.07,"bytes_per_snapshot":61140992.0,"last_snapshot_bytes":61140992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+replayingreplaying, {"bytes_per_second":2205149.87,"bytes_per_snapshot":66154496.0,"last_snapshot_bytes":66154496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+replayingreplaying, {"bytes_per_second":3145113.6,"bytes_per_snapshot":62902272.0,"last_snapshot_bytes":62902272,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+replayingreplaying, {"bytes_per_second":3151052.8,"bytes_per_snapshot":63021056.0,"last_snapshot_bytes":63021056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+replayingreplaying, {"bytes_per_second":2124185.6,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary9814425c6-1326-454a-85d2-08f9f100d357up+replayingreplaying, {"bytes_per_second":3066675.2,"bytes_per_snapshot":61333504.0,"last_snapshot_bytes":61333504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17' 2024-09-18T00:20:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:20:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:20:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:20:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:20:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:20:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-18T00:20:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:20:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:20:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-18T00:20:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-18T00:20:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:20:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-18T00:20:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:20:21.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:21.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:20:21.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary1 2024-09-18T00:20:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary1 2024-09-18T00:20:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-18T00:20:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-18T00:20:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:20:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:20:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-18T00:20:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:20:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:20:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-18T00:20:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:20:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:20:22.158 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:20:22.155+0000 7f9bfab49640 -1 reset not still connected to 0x55cd12ca15f0 2024-09-18T00:20:22.161 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:20:22.157+0000 7f9bfab49640 -1 reset not still connected to 0x55cd18797ba0 2024-09-18T00:20:22.161 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:20:22.158+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1d5081a0 2024-09-18T00:20:22.161 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:20:22.158+0000 7f9bfab49640 -1 reset not still connected to 0x55cd2bf77110 2024-09-18T00:20:22.161 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:20:22.158+0000 7f9bfab49640 -1 reset not still connected to 0x55cd39c900d0 2024-09-18T00:20:22.162 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:20:22.158+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3b8e7450 2024-09-18T00:20:22.162 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:20:22.158+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3b8e7790 2024-09-18T00:20:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:20:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d64cf4b9-98c4-4f70-a161-76e8c07ce938 2024-09-18T00:20:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:20:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3159859.2,"bytes_per_snapshot":63197184.0,"last_snapshot_bytes":63197184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"} 2024-09-18T00:20:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:20:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:17 2024-09-18T00:20:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:20:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:20:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:20:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:20:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:17' 2024-09-18T00:20:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:20:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:20:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:20:22.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3159859.2,"bytes_per_snapshot":63197184.0,"last_snapshot_bytes":63197184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"} 2024-09-18T00:20:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:20:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:22.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:20:22.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:20:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:20:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:20:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:20:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-18 00:20:17' 2024-09-18T00:20:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:20:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:20:17' 2024-09-18T00:20:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:20:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK942634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":2106845.87,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary2d64cf4b9-98c4-4f70-a161-76e8c07ce938up+replayingreplaying, {"bytes_per_second":3159859.2,"bytes_per_snapshot":63197184.0,"last_snapshot_bytes":63197184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+replayingreplaying, {"bytes_per_second":3337625.6,"bytes_per_snapshot":66752512.0,"last_snapshot_bytes":66752512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+replayingreplaying, {"bytes_per_second":2038033.07,"bytes_per_snapshot":61140992.0,"last_snapshot_bytes":61140992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+replayingreplaying, {"bytes_per_second":2205149.87,"bytes_per_snapshot":66154496.0,"last_snapshot_bytes":66154496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+replayingreplaying, {"bytes_per_second":3145113.6,"bytes_per_snapshot":62902272.0,"last_snapshot_bytes":62902272,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+replayingreplaying, {"bytes_per_second":3151052.8,"bytes_per_snapshot":63021056.0,"last_snapshot_bytes":63021056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+replayingreplaying, {"bytes_per_second":2124185.6,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17image-primary9814425c6-1326-454a-85d2-08f9f100d357up+replayingreplaying, {"bytes_per_second":3066675.2,"bytes_per_snapshot":61333504.0,"last_snapshot_bytes":61333504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:17' 2024-09-18T00:20:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:20:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:20:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:20:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:20:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:20:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-18T00:20:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:20:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:20:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-18T00:20:22.335 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:20:22.332+0000 7f6acfc5f640 -1 reset not still connected to 0x555c0dfe2270 2024-09-18T00:20:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-18T00:20:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:20:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-18T00:20:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:20:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:20:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary2 2024-09-18T00:20:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary2 2024-09-18T00:20:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-18T00:20:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-18T00:20:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:20:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:20:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-18T00:20:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:20:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:20:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-18T00:20:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:20:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:20:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:20:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 5aa63b87-d45e-4d8e-a667-5dafb6b4ef7c 2024-09-18T00:20:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:20:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3337625.6,"bytes_per_snapshot":66752512.0,"last_snapshot_bytes":66752512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"} 2024-09-18T00:20:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:20:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:17 2024-09-18T00:20:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:20:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:20:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:20:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:20:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:22' 2024-09-18T00:20:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:20:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:20:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:20:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3337625.6,"bytes_per_snapshot":66752512.0,"last_snapshot_bytes":66752512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"} 2024-09-18T00:20:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:20:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:20:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:20:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:20:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:20:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:20:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-18 00:20:17' 2024-09-18T00:20:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:20:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:20:17' 2024-09-18T00:20:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:20:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK842634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":2106845.87,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:22image-primary35aa63b87-d45e-4d8e-a667-5dafb6b4ef7cup+replayingreplaying, {"bytes_per_second":3337625.6,"bytes_per_snapshot":66752512.0,"last_snapshot_bytes":66752512,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:22image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+replayingreplaying, {"bytes_per_second":2038033.07,"bytes_per_snapshot":61140992.0,"last_snapshot_bytes":61140992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:22image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+replayingreplaying, {"bytes_per_second":2205149.87,"bytes_per_snapshot":66154496.0,"last_snapshot_bytes":66154496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:22image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+replayingreplaying, {"bytes_per_second":3145113.6,"bytes_per_snapshot":62902272.0,"last_snapshot_bytes":62902272,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:22image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+replayingreplaying, {"bytes_per_second":3151052.8,"bytes_per_snapshot":63021056.0,"last_snapshot_bytes":63021056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:22image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+replayingreplaying, {"bytes_per_second":2124185.6,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:22image-primary9814425c6-1326-454a-85d2-08f9f100d357up+replayingreplaying, {"bytes_per_second":3066675.2,"bytes_per_snapshot":61333504.0,"last_snapshot_bytes":61333504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:22' 2024-09-18T00:20:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:20:25.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:20:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:20:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:20:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:20:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-18T00:20:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:20:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:20:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-18T00:20:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-18T00:20:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:20:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-18T00:20:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:20:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:20:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary3 2024-09-18T00:20:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary3 2024-09-18T00:20:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-18T00:20:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-18T00:20:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:20:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:20:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-18T00:20:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:20:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:20:28.013 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:20:28.010+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3b8e7450 2024-09-18T00:20:28.013 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:20:28.010+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3b8e7790 2024-09-18T00:20:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-18T00:20:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:20:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-18T00:20:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:20:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 98347238-de9a-416e-80a1-1bb76d54ed82 2024-09-18T00:20:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:20:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2038033.07,"bytes_per_snapshot":61140992.0,"last_snapshot_bytes":61140992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"} 2024-09-18T00:20:28.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:20:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:17 2024-09-18T00:20:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:20:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:20:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:20:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:20:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:27' 2024-09-18T00:20:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:20:28.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:20:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:20:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2038033.07,"bytes_per_snapshot":61140992.0,"last_snapshot_bytes":61140992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"} 2024-09-18T00:20:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:20:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:20:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:28.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:20:28.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:20:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:20:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:20:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-18 00:20:17' 2024-09-18T00:20:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:20:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:20:17' 2024-09-18T00:20:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:20:28.322 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:20:28.320+0000 7f6acfc5f640 -1 reset not still connected to 0x555c0ccd44e0 2024-09-18T00:20:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK742634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":2106845.87,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:27image-primary498347238-de9a-416e-80a1-1bb76d54ed82up+replayingreplaying, {"bytes_per_second":2038033.07,"bytes_per_snapshot":61140992.0,"last_snapshot_bytes":61140992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:27image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+replayingreplaying, {"bytes_per_second":2205149.87,"bytes_per_snapshot":66154496.0,"last_snapshot_bytes":66154496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:27image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+replayingreplaying, {"bytes_per_second":3145113.6,"bytes_per_snapshot":62902272.0,"last_snapshot_bytes":62902272,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:27image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+replayingreplaying, {"bytes_per_second":3151052.8,"bytes_per_snapshot":63021056.0,"last_snapshot_bytes":63021056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:27image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+replayingreplaying, {"bytes_per_second":2124185.6,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:27image-primary9814425c6-1326-454a-85d2-08f9f100d357up+replayingreplaying, {"bytes_per_second":3066675.2,"bytes_per_snapshot":61333504.0,"last_snapshot_bytes":61333504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:27' 2024-09-18T00:20:28.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:20:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:20:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:20:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:20:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:20:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-18T00:20:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:20:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:20:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-18T00:20:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-18T00:20:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:20:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-18T00:20:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:20:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:20:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary4 2024-09-18T00:20:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary4 2024-09-18T00:20:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-18T00:20:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-18T00:20:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:20:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:20:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-18T00:20:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:20:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:20:31.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-18T00:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-18T00:20:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:20:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: eff149be-58c6-4f92-96d5-04d6dc49412e 2024-09-18T00:20:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:20:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2205149.87,"bytes_per_snapshot":66154496.0,"last_snapshot_bytes":66154496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"} 2024-09-18T00:20:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:20:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:17 2024-09-18T00:20:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:20:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:20:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:20:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:20:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:27' 2024-09-18T00:20:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:20:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:20:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:20:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2205149.87,"bytes_per_snapshot":66154496.0,"last_snapshot_bytes":66154496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"} 2024-09-18T00:20:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:20:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:20:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:20:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:20:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:20:31.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-18 00:20:17' 2024-09-18T00:20:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:20:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:20:17' 2024-09-18T00:20:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:20:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK642634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":2106845.87,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:27image-primary5eff149be-58c6-4f92-96d5-04d6dc49412eup+replayingreplaying, {"bytes_per_second":2205149.87,"bytes_per_snapshot":66154496.0,"last_snapshot_bytes":66154496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:27image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+replayingreplaying, {"bytes_per_second":3145113.6,"bytes_per_snapshot":62902272.0,"last_snapshot_bytes":62902272,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:27image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+replayingreplaying, {"bytes_per_second":3151052.8,"bytes_per_snapshot":63021056.0,"last_snapshot_bytes":63021056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:27image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+replayingreplaying, {"bytes_per_second":2124185.6,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:27image-primary9814425c6-1326-454a-85d2-08f9f100d357up+replayingreplaying, {"bytes_per_second":3066675.2,"bytes_per_snapshot":61333504.0,"last_snapshot_bytes":61333504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:27' 2024-09-18T00:20:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:20:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:20:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:20:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:20:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:31.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:20:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-18T00:20:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:20:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:20:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-18T00:20:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-18T00:20:33.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:20:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-18T00:20:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:33.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:20:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:20:33.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary5 2024-09-18T00:20:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary5 2024-09-18T00:20:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-18T00:20:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-18T00:20:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:20:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:20:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-18T00:20:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:20:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:20:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-18T00:20:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:20:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:20:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:20:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6 2024-09-18T00:20:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:20:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3145113.6,"bytes_per_snapshot":62902272.0,"last_snapshot_bytes":62902272,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"} 2024-09-18T00:20:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:20:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:17 2024-09-18T00:20:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:20:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:20:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:20:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:20:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:32' 2024-09-18T00:20:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:20:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:20:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:20:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3145113.6,"bytes_per_snapshot":62902272.0,"last_snapshot_bytes":62902272,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"} 2024-09-18T00:20:34.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:20:34.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:34.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:20:34.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:20:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:20:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:20:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:20:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-18 00:20:17' 2024-09-18T00:20:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:20:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:20:17' 2024-09-18T00:20:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:20:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK542634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":2106845.87,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:32image-primary6ef4151e6-1a12-4ca9-9ad3-7b9252ad05a6up+replayingreplaying, {"bytes_per_second":3145113.6,"bytes_per_snapshot":62902272.0,"last_snapshot_bytes":62902272,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:32image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+replayingreplaying, {"bytes_per_second":3151052.8,"bytes_per_snapshot":63021056.0,"last_snapshot_bytes":63021056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:32image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+replayingreplaying, {"bytes_per_second":2124185.6,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:32image-primary9814425c6-1326-454a-85d2-08f9f100d357up+replayingreplaying, {"bytes_per_second":3066675.2,"bytes_per_snapshot":61333504.0,"last_snapshot_bytes":61333504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:32' 2024-09-18T00:20:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:20:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:20:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:20:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:20:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:20:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-18T00:20:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:20:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:20:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-18T00:20:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-18T00:20:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:20:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-18T00:20:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:20:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:20:36.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary6 2024-09-18T00:20:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary6 2024-09-18T00:20:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-18T00:20:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-18T00:20:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:20:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:20:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-18T00:20:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:20:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:20:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-18T00:20:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:20:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:20:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:20:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 02b583ab-f4b1-4c02-b3ba-6dd65cd3316c 2024-09-18T00:20:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:20:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3151052.8,"bytes_per_snapshot":63021056.0,"last_snapshot_bytes":63021056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"} 2024-09-18T00:20:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:20:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:17 2024-09-18T00:20:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:20:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:20:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:20:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:20:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:32' 2024-09-18T00:20:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:20:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:20:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:20:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3151052.8,"bytes_per_snapshot":63021056.0,"last_snapshot_bytes":63021056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"} 2024-09-18T00:20:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:20:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:20:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:20:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:20:37.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:20:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:20:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-18 00:20:17' 2024-09-18T00:20:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:20:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:20:17' 2024-09-18T00:20:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:20:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":2106845.87,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:32image-primary702b583ab-f4b1-4c02-b3ba-6dd65cd3316cup+replayingreplaying, {"bytes_per_second":3151052.8,"bytes_per_snapshot":63021056.0,"last_snapshot_bytes":63021056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:32image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+replayingreplaying, {"bytes_per_second":2124185.6,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:32image-primary9814425c6-1326-454a-85d2-08f9f100d357up+replayingreplaying, {"bytes_per_second":3066675.2,"bytes_per_snapshot":61333504.0,"last_snapshot_bytes":61333504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:32' 2024-09-18T00:20:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:20:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:20:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:20:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:20:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:20:37.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-18T00:20:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:20:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:20:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-18T00:20:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-18T00:20:38.533 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:20:38.531+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:20:38.534 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:20:38.531+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-18T00:20:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:20:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-18T00:20:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:20:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:20:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary7 2024-09-18T00:20:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary7 2024-09-18T00:20:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-18T00:20:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-18T00:20:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:20:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:20:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-18T00:20:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:20:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:20:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-18T00:20:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:20:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:40.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:20:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:20:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a827873c-19b1-443f-b747-bed14c3f54c3 2024-09-18T00:20:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:20:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2124185.6,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"} 2024-09-18T00:20:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:20:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:17 2024-09-18T00:20:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:20:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:20:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:20:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:20:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:37' 2024-09-18T00:20:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:20:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:20:40.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:20:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2124185.6,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"} 2024-09-18T00:20:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:20:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:20:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:20:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:20:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:20:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:20:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-18 00:20:17' 2024-09-18T00:20:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:20:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:20:17' 2024-09-18T00:20:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:20:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK342634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":2106845.87,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:37image-primary8a827873c-19b1-443f-b747-bed14c3f54c3up+replayingreplaying, {"bytes_per_second":2124185.6,"bytes_per_snapshot":63725568.0,"last_snapshot_bytes":63725568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:37image-primary9814425c6-1326-454a-85d2-08f9f100d357up+replayingreplaying, {"bytes_per_second":3066675.2,"bytes_per_snapshot":61333504.0,"last_snapshot_bytes":61333504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:37' 2024-09-18T00:20:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:20:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:20:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:20:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:20:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:20:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-18T00:20:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:20:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:20:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-18T00:20:40.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-18T00:20:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:20:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-18T00:20:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:20:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:20:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary8 2024-09-18T00:20:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary8 2024-09-18T00:20:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-18T00:20:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-18T00:20:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:20:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:20:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-18T00:20:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:20:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:20:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-18T00:20:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:20:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:43.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:20:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:20:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 814425c6-1326-454a-85d2-08f9f100d357 2024-09-18T00:20:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:20:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3066675.2,"bytes_per_snapshot":61333504.0,"last_snapshot_bytes":61333504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"} 2024-09-18T00:20:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:20:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:17 2024-09-18T00:20:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:20:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:20:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:20:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:20:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:42' 2024-09-18T00:20:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:20:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:20:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:20:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3066675.2,"bytes_per_snapshot":61333504.0,"last_snapshot_bytes":61333504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"} 2024-09-18T00:20:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:20:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:20:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:20:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:20:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:20:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:20:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-18 00:20:17' 2024-09-18T00:20:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:20:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:20:17' 2024-09-18T00:20:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:20:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK242634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":2106845.87,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:42image-primary9814425c6-1326-454a-85d2-08f9f100d357up+replayingreplaying, {"bytes_per_second":3066675.2,"bytes_per_snapshot":61333504.0,"last_snapshot_bytes":61333504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618788,"remote_snapshot_timestamp":1726618788,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:42' 2024-09-18T00:20:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:20:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:20:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:20:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:20:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:20:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-18T00:20:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:20:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:20:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-18T00:20:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-18T00:20:45.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:20:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-18T00:20:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:20:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:20:45.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary9 2024-09-18T00:20:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary9 2024-09-18T00:20:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-18T00:20:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-18T00:20:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:20:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:20:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-18T00:20:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:20:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:45.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:45.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:45.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:20:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-18T00:20:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:20:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:20:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:20:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:20:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:20:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d2b68ce6-80c4-4597-8006-4885dadbe8d9 2024-09-18T00:20:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:20:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2106845.87,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"} 2024-09-18T00:20:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:20:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:17 2024-09-18T00:20:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:20:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:20:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:20:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:20:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:20:42' 2024-09-18T00:20:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:20:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:20:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:20:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2106845.87,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"} 2024-09-18T00:20:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:20:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:20:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:20:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:20:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:20:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:20:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-18 00:20:17' 2024-09-18T00:20:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:20:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:20:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:20:17' 2024-09-18T00:20:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:20:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:20:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:20:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary10d2b68ce6-80c4-4597-8006-4885dadbe8d9up+replayingreplaying, {"bytes_per_second":2106845.87,"bytes_per_snapshot":63205376.0,"last_snapshot_bytes":63205376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726618780,"remote_snapshot_timestamp":1726618780,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:20:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:20:42' 2024-09-18T00:20:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:20:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:20:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:20:17' 2024-09-18T00:20:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:20:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:20:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:20:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:20:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:20:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-18T00:20:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:20:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:20:46.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-18T00:20:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-18T00:20:48.025 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:20:48.023+0000 7fd413089640 -1 reset not still connected to 0x5604aaecb520 2024-09-18T00:20:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:20:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-18T00:20:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:20:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:20:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:20:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary10 2024-09-18T00:20:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:48.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary10 2024-09-18T00:20:48.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..10} 2024-09-18T00:20:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS=() 2024-09-18T00:20:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS=() 2024-09-18T00:20:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS=() 2024-09-18T00:20:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS=() 2024-09-18T00:20:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ RET=0 2024-09-18T00:20:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-18T00:20:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary1 2024-09-18T00:20:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-18T00:20:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:20:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:20:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-18T00:20:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:20:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:48.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:20:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:20:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:20:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-18T00:20:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:20:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:20:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-18T00:20:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:20:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-18T00:20:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:20:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-18T00:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary1': 2024-09-18T00:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 308e6d2741b4 2024-09-18T00:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.308e6d2741b4 2024-09-18T00:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:20:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:20:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:20:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:20:48 2024 2024-09-18T00:20:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:20:48 2024 2024-09-18T00:20:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:20:48 2024 2024-09-18T00:20:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:20:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:20:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 8ae5cefc-b421-4d99-a945-ce61f7358a1b 2024-09-18T00:20:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:20:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-18T00:20:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd0 2024-09-18T00:20:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:20:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-18T00:20:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:20:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:20:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:20:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 5d2e164f-e208-4744-b1b1-46b7177e1abb 2024-09-18T00:20:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:20:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:20:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:20:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:20:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:20:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:20:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary1 2024-09-18T00:20:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-18T00:20:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:20:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary1 2024-09-18T00:20:49.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:20:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-18T00:20:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:20:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-18T00:20:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary2 2024-09-18T00:20:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary1 2024-09-18T00:20:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-18T00:20:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:20:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:20:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:20:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary1 2024-09-18T00:20:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:20:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-18T00:20:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-18T00:20:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:20:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:20:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:20:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:20:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:20:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:20:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-18T00:20:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:20:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:20:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-18T00:20:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:20:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:49.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-18T00:20:49.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:20:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-18T00:20:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:20:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2316 2024-09-18T00:20:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-18T00:20:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary2': 2024-09-18T00:20:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:20:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:20:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:20:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 309b79c0fb3 2024-09-18T00:20:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.309b79c0fb3 2024-09-18T00:20:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:20:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:20:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:20:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:20:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:20:49 2024 2024-09-18T00:20:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:20:49 2024 2024-09-18T00:20:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:20:49 2024 2024-09-18T00:20:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:20:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:20:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 3a078d42-4f72-4a33-8636-ba1da94d8aa6 2024-09-18T00:20:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:20:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:20:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:20:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-18T00:20:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd1 2024-09-18T00:20:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:20:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-18T00:20:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:20:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:20:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:20:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 48f69012-f4d6-4d59-9028-d1905119b09f 2024-09-18T00:20:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:20:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:20:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:20:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:20:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:20:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:20:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:49.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary2 2024-09-18T00:20:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-18T00:20:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:20:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary2 2024-09-18T00:20:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:20:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-18T00:20:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:20:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:20:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-18T00:20:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary2 2024-09-18T00:20:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary3 2024-09-18T00:20:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-18T00:20:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:20:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:20:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary2 2024-09-18T00:20:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:20:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-18T00:20:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:20:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:20:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-18T00:20:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:20:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:20:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:20:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:20:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-18T00:20:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:20:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:20:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-18T00:20:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:20:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-18T00:20:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:20:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-18T00:20:50.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:20:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2318 2024-09-18T00:20:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-18T00:20:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary3': 2024-09-18T00:20:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:20:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:20:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:20:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 30aa975a7386 2024-09-18T00:20:50.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.30aa975a7386 2024-09-18T00:20:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:20:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:20:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:20:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:20:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:20:50 2024 2024-09-18T00:20:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:20:50 2024 2024-09-18T00:20:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:20:50 2024 2024-09-18T00:20:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:20:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:20:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 1745ddfe-ba46-4c43-8c01-3b0b11ca89ce 2024-09-18T00:20:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:20:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:20:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:20:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-18T00:20:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd2 2024-09-18T00:20:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:20:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-18T00:20:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:20:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:20:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:20:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: be66b276-fd66-4e7c-aab1-2ea986a68f40 2024-09-18T00:20:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:20:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:20:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:20:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:20:50.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary3 2024-09-18T00:20:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-18T00:20:51.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:20:51.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary3 2024-09-18T00:20:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:20:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-18T00:20:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:20:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:20:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:20:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary3 2024-09-18T00:20:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-18T00:20:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary4 2024-09-18T00:20:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:20:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-18T00:20:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:20:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary3 2024-09-18T00:20:51.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:20:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-18T00:20:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:20:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-18T00:20:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:20:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:20:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:20:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:20:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-18T00:20:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:20:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:20:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-18T00:20:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:20:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-18T00:20:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:20:51.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-18T00:20:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:20:51.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-18T00:20:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2320 2024-09-18T00:20:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary4': 2024-09-18T00:20:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:20:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:20:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:20:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 30b965a6d312 2024-09-18T00:20:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.30b965a6d312 2024-09-18T00:20:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:20:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:20:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:20:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:20:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:20:51 2024 2024-09-18T00:20:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:20:51 2024 2024-09-18T00:20:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:20:51 2024 2024-09-18T00:20:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:20:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:20:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: ce11166a-ff96-4612-8765-db39b0dba9f3 2024-09-18T00:20:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:20:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:20:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:20:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-18T00:20:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd3 2024-09-18T00:20:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:20:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-18T00:20:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:20:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:20:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:20:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 6b7fa818-54cb-4a79-a9df-31199df92f39 2024-09-18T00:20:52.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:20:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:20:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:20:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:20:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:20:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:20:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:52.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary4 2024-09-18T00:20:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-18T00:20:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:20:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary4 2024-09-18T00:20:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:20:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-18T00:20:52.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:20:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:20:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:20:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-18T00:20:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary5 2024-09-18T00:20:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:20:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary4 2024-09-18T00:20:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-18T00:20:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:20:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:20:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary4 2024-09-18T00:20:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:20:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-18T00:20:52.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-18T00:20:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:20:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:20:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:20:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:20:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-18T00:20:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:20:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:20:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-18T00:20:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:20:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:52.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-18T00:20:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:20:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-18T00:20:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:20:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:20:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:20:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:20:52.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-18T00:20:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2322 2024-09-18T00:20:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary5': 2024-09-18T00:20:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:20:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:20:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:20:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 30c8b5b1783 2024-09-18T00:20:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.30c8b5b1783 2024-09-18T00:20:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:20:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:20:52.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:20:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:20:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:20:52 2024 2024-09-18T00:20:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:20:52 2024 2024-09-18T00:20:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:20:52 2024 2024-09-18T00:20:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:20:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:20:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: aa5f09f2-9a9e-4dac-b2c1-219170bfb686 2024-09-18T00:20:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:20:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:20:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:20:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-18T00:20:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd4 2024-09-18T00:20:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:20:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-18T00:20:53.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:20:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:20:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:20:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 66c51a73-4e32-4bd1-951f-e6ed353eebda 2024-09-18T00:20:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:20:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:20:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:20:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:20:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:20:53.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:20:53.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:53.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary5 2024-09-18T00:20:53.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-18T00:20:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:20:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary5 2024-09-18T00:20:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:20:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-18T00:20:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:20:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:20:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:20:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary5 2024-09-18T00:20:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-18T00:20:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary6 2024-09-18T00:20:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:20:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-18T00:20:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:20:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary5 2024-09-18T00:20:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:20:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-18T00:20:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:20:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-18T00:20:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:20:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:20:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:20:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:20:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-18T00:20:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:20:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:20:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-18T00:20:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:20:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-18T00:20:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:20:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-18T00:20:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:20:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:20:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:20:53.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:20:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2325 2024-09-18T00:20:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-18T00:20:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary6': 2024-09-18T00:20:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:20:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:20:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:20:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 30d951f8cff0 2024-09-18T00:20:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.30d951f8cff0 2024-09-18T00:20:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:20:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:20:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:20:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:20:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:20:53 2024 2024-09-18T00:20:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:20:53 2024 2024-09-18T00:20:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:20:53 2024 2024-09-18T00:20:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:20:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:20:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 1af2badd-a447-4ef2-be7f-7f2e9a23b3c7 2024-09-18T00:20:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:20:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:20:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:20:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-18T00:20:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd5 2024-09-18T00:20:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:20:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-18T00:20:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:20:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:20:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:20:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 30638a32-a3e1-4f51-bab4-5164b2f846a8 2024-09-18T00:20:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:20:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:20:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:20:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:20:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:20:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:20:54.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary6 2024-09-18T00:20:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-18T00:20:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:20:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary6 2024-09-18T00:20:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:20:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-18T00:20:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:20:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:20:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:20:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary6 2024-09-18T00:20:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-18T00:20:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary7 2024-09-18T00:20:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:20:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-18T00:20:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:20:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary6 2024-09-18T00:20:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-18T00:20:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:20:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:20:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-18T00:20:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:54.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:20:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:20:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:20:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:20:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-18T00:20:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:20:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:20:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-18T00:20:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:20:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-18T00:20:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:20:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-18T00:20:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2324 2024-09-18T00:20:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:54.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:20:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2328 2024-09-18T00:20:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-18T00:20:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:20:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:20:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:20:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary7': 2024-09-18T00:20:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:20:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:20:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:20:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 30e8766f8eec 2024-09-18T00:20:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.30e8766f8eec 2024-09-18T00:20:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:20:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:20:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:20:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:20:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:20:54 2024 2024-09-18T00:20:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:20:54 2024 2024-09-18T00:20:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:20:54 2024 2024-09-18T00:20:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:20:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:20:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9ee95762-996e-4381-9c29-8aeddf45ccc6 2024-09-18T00:20:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:20:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:20:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:20:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-18T00:20:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd6 2024-09-18T00:20:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:20:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-18T00:20:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:20:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:20:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:20:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 2c15d44c-7ca0-4a20-8134-0e2a26385098 2024-09-18T00:20:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:20:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:20:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:20:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:20:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:20:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:20:55.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary7 2024-09-18T00:20:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-18T00:20:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:20:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary7 2024-09-18T00:20:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:20:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-18T00:20:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:20:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:20:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-18T00:20:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary8 2024-09-18T00:20:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:20:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary7 2024-09-18T00:20:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-18T00:20:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:20:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:20:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:20:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary7 2024-09-18T00:20:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:20:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-18T00:20:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-18T00:20:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:20:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:20:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:20:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:20:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-18T00:20:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:20:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:20:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-18T00:20:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:20:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-18T00:20:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:20:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-18T00:20:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:20:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:20:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:20:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:20:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2331 2024-09-18T00:20:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-18T00:20:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary8': 2024-09-18T00:20:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:20:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:20:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:20:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 30f7b70b681e 2024-09-18T00:20:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.30f7b70b681e 2024-09-18T00:20:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:20:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:20:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:20:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:20:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:20:55 2024 2024-09-18T00:20:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:20:55 2024 2024-09-18T00:20:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:20:55 2024 2024-09-18T00:20:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:20:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:20:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: c5dfa332-f995-41a3-a007-b88c3f9ab788 2024-09-18T00:20:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:20:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:20:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:20:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-18T00:20:56.048 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:20:56.040+0000 7f0114255640 -1 reset not still connected to 0x557d09ccc750 2024-09-18T00:20:56.048 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:20:56.040+0000 7f0114255640 -1 reset not still connected to 0x557d09ccdd40 2024-09-18T00:20:56.048 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:20:56.040+0000 7f0114255640 -1 reset not still connected to 0x557d09df6b60 2024-09-18T00:20:56.048 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:20:56.040+0000 7f0114255640 -1 reset not still connected to 0x557d09df6d00 2024-09-18T00:20:56.048 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:20:56.040+0000 7f0114255640 -1 reset not still connected to 0x557d0a065ee0 2024-09-18T00:20:56.048 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:20:56.040+0000 7f0114255640 -1 reset not still connected to 0x557d0b464750 2024-09-18T00:20:56.048 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:20:56.040+0000 7f0114255640 -1 reset not still connected to 0x557d141fe340 2024-09-18T00:20:56.048 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:20:56.040+0000 7f0114255640 -1 reset not still connected to 0x557d14d3ca90 2024-09-18T00:20:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd7 2024-09-18T00:20:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:20:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-18T00:20:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:20:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:20:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:20:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: ddcb9546-cc3e-47db-b241-4c33752a4aa7 2024-09-18T00:20:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:20:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:20:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:20:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:20:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:20:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:20:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary8 2024-09-18T00:20:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-18T00:20:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:20:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary8 2024-09-18T00:20:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:20:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-18T00:20:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:20:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:20:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:20:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:20:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:20:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-18T00:20:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary8 2024-09-18T00:20:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary9 2024-09-18T00:20:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-18T00:20:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary8 2024-09-18T00:20:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:20:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-18T00:20:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:20:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-18T00:20:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:20:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:20:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:20:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:20:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-18T00:20:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:20:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:20:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-18T00:20:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:20:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-18T00:20:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:20:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-18T00:20:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2327 2024-09-18T00:20:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2330 2024-09-18T00:20:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:20:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:20:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:20:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:20:56.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2334 2024-09-18T00:20:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-18T00:20:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary9': 2024-09-18T00:20:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:20:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:20:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:20:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 31076591d9be 2024-09-18T00:20:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.31076591d9be 2024-09-18T00:20:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:20:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:20:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:20:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:20:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:20:56 2024 2024-09-18T00:20:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:20:56 2024 2024-09-18T00:20:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:20:56 2024 2024-09-18T00:20:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:20:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:20:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: d08650b4-8cb9-4025-8199-1719a7f72860 2024-09-18T00:20:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:20:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:20:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:20:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-18T00:20:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd8 2024-09-18T00:20:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:20:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-18T00:20:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:20:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:20:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:20:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: e00c416b-4e30-4b76-b1b1-6201fe4b99bd 2024-09-18T00:20:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:20:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:20:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:20:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:20:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:20:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:20:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary9 2024-09-18T00:20:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-18T00:20:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:20:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary9 2024-09-18T00:20:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:20:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-18T00:20:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:20:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:20:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:20:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-18T00:20:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary10 2024-09-18T00:20:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary9 2024-09-18T00:20:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:20:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-18T00:20:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:20:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:20:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:20:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:20:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary9 2024-09-18T00:20:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-18T00:20:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-18T00:20:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:20:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:20:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:20:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:20:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:20:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-18T00:20:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:20:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:20:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:20:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-18T00:20:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:20:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:20:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-18T00:20:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:20:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-18T00:20:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:20:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:20:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:20:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2333 2024-09-18T00:20:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:20:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:20:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:20:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:20:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2337 2024-09-18T00:20:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-18T00:20:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary10': 2024-09-18T00:20:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:20:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:20:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:20:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 3119131550b5 2024-09-18T00:20:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.3119131550b5 2024-09-18T00:20:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:20:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:20:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:20:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:20:58.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:20:57 2024 2024-09-18T00:20:58.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:20:57 2024 2024-09-18T00:20:58.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:20:57 2024 2024-09-18T00:20:58.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:20:58.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:20:58.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 10dacb73-f415-417d-8ed1-6bdb0fc5fcf2 2024-09-18T00:20:58.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:20:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:20:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:20:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-18T00:20:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd9 2024-09-18T00:20:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:20:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-18T00:20:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:20:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:20:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:20:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: aee60858-aeee-4c2f-b726-e02d0dd082e4 2024-09-18T00:20:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:20:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:20:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:20:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:20:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:20:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:20:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:20:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary10 2024-09-18T00:20:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-18T00:20:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:20:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary10 2024-09-18T00:20:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:20:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-18T00:20:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:20:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:20:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:20:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:20:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary10 2024-09-18T00:20:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:20:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 118603 2024-09-18T00:20:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:20:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary10 2024-09-18T00:20:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-18T00:20:58.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:20:58.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-18T00:20:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2336 2024-09-18T00:20:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2341 2024-09-18T00:20:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:20:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:20:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:20:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:59.198 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:20:59.196+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:20:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2339 2024-09-18T00:20:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2340 2024-09-18T00:20:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:20:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:20:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:20:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:20:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:20:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:20:59.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:20:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:20:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:20:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:21:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:21:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:21:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:21:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2342 2024-09-18T00:21:00.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2343 2024-09-18T00:21:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:21:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:21:00.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:21:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:21:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:21:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:21:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2345 2024-09-18T00:21:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:21:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:21:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:21:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:21:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:21:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:21:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:21:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:21:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:21:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:21:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2346 2024-09-18T00:21:02.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2347 2024-09-18T00:21:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2344 2024-09-18T00:21:02.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:03.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2348 2024-09-18T00:21:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2349 2024-09-18T00:21:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:21:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:21:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:21:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:21:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:21:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:21:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2351 2024-09-18T00:21:04.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2350 2024-09-18T00:21:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:21:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:21:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:21:04.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:04.977 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:21:04.974+0000 7f9bfab49640 -1 reset not still connected to 0x55cd2b33a1a0 2024-09-18T00:21:04.977 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:21:04.974+0000 7f9bfab49640 -1 reset not still connected to 0x55cd434ae0d0 2024-09-18T00:21:05.238 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:21:05.236+0000 7f0114255640 -1 reset not still connected to 0x557d0a065ee0 2024-09-18T00:21:05.238 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:21:05.236+0000 7f0114255640 -1 reset not still connected to 0x557d0b464750 2024-09-18T00:21:05.238 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:21:05.236+0000 7f0114255640 -1 reset not still connected to 0x557d141fe340 2024-09-18T00:21:05.238 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:21:05.236+0000 7f0114255640 -1 reset not still connected to 0x557d14d3ca90 2024-09-18T00:21:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2352 2024-09-18T00:21:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2353 2024-09-18T00:21:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:21:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:21:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:21:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:21:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:21:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:21:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:21:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:21:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:21:06.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2354 2024-09-18T00:21:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:21:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:21:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:21:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:06.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2356 2024-09-18T00:21:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:21:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:21:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:21:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:21:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:21:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:21:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:21:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:21:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:21:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2357 2024-09-18T00:21:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2355 2024-09-18T00:21:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:21:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:21:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:21:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2358 2024-09-18T00:21:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2359 2024-09-18T00:21:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:21:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:21:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:21:08.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:09.331 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:21:09.330+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-18T00:21:09.332 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:21:09.330+0000 7f9bfab49640 -1 reset not still connected to 0x55cd13254750 2024-09-18T00:21:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2361 2024-09-18T00:21:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2360 2024-09-18T00:21:09.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:09.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:21:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:21:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:21:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:21:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:21:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:21:09.954 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:21:09.952+0000 7f0114255640 -1 reset not still connected to 0x557d09ccc4e0 2024-09-18T00:21:09.954 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:21:09.952+0000 7f0114255640 -1 reset not still connected to 0x557d09ccc750 2024-09-18T00:21:09.956 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:21:09.953+0000 7f0114255640 -1 reset not still connected to 0x557d09ccdd40 2024-09-18T00:21:09.956 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:21:09.953+0000 7f0114255640 -1 reset not still connected to 0x557d09df6d00 2024-09-18T00:21:09.956 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:21:09.953+0000 7f0114255640 -1 reset not still connected to 0x557d0a065ee0 2024-09-18T00:21:09.957 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:21:09.953+0000 7f0114255640 -1 reset not still connected to 0x557d0b464750 2024-09-18T00:21:09.957 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:21:09.953+0000 7f0114255640 -1 reset not still connected to 0x557d141fe340 2024-09-18T00:21:09.957 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:21:09.953+0000 7f0114255640 -1 reset not still connected to 0x557d14d3ca90 2024-09-18T00:21:10.199 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:21:10.197+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:21:10.199 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:21:10.197+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-18T00:21:10.222 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:21:10.216+0000 7f9bfab49640 -1 reset not still connected to 0x55cd13254750 2024-09-18T00:21:10.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2362 2024-09-18T00:21:10.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2363 2024-09-18T00:21:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:10.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:21:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:21:10.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:21:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:21:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:21:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:21:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2365 2024-09-18T00:21:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2364 2024-09-18T00:21:11.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:21:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:21:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:21:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:21:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:21:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:21:12.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:21:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:21:12.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:21:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:21:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:21:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:21:12.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2366 2024-09-18T00:21:12.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2367 2024-09-18T00:21:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:12.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2369 2024-09-18T00:21:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2368 2024-09-18T00:21:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:21:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:21:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:21:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:21:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:21:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:21:13.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:14.080 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:21:14.078+0000 7f0114255640 -1 reset not still connected to 0x557d0b464750 2024-09-18T00:21:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2370 2024-09-18T00:21:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2371 2024-09-18T00:21:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:21:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:21:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:21:14.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:21:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:21:14.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:21:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2372 2024-09-18T00:21:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:21:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:21:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:21:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:21:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:21:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:21:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2375 2024-09-18T00:21:16.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2374 2024-09-18T00:21:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2373 2024-09-18T00:21:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:21:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:21:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:21:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:21:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:21:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:21:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2377 2024-09-18T00:21:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2376 2024-09-18T00:21:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:21:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:21:17.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:21:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:21:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:21:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:21:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2379 2024-09-18T00:21:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2378 2024-09-18T00:21:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:21:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:21:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:21:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2380 2024-09-18T00:21:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:21:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:21:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:21:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:21:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:21:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:21:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:21:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:21:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:21:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2381 2024-09-18T00:21:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2382 2024-09-18T00:21:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:21:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:21:20.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:21:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:21:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:21:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:21:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2384 2024-09-18T00:21:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2383 2024-09-18T00:21:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2385 2024-09-18T00:21:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:21:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:21:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:21:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:21:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:21:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:21:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2386 2024-09-18T00:21:22.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2387 2024-09-18T00:21:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:21:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:21:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:21:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:23.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2389 2024-09-18T00:21:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2388 2024-09-18T00:21:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:21:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:21:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:21:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:21:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:21:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:21:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:21:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:21:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:21:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2390 2024-09-18T00:21:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2391 2024-09-18T00:21:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:21:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:21:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:21:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:21:24.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:21:24.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:21:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2393 2024-09-18T00:21:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2392 2024-09-18T00:21:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:21:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:21:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:21:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:21:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:21:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:21:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2394 2024-09-18T00:21:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2395 2024-09-18T00:21:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:21:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:21:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:21:26.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:21:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:21:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:21:26.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:21:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:21:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:21:27.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2397 2024-09-18T00:21:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2396 2024-09-18T00:21:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:21:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:21:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:21:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2399 2024-09-18T00:21:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2398 2024-09-18T00:21:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:21:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:21:28.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:21:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:21:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:21:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:21:28.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2400 2024-09-18T00:21:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2401 2024-09-18T00:21:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:29.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:21:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:21:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:21:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:21:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:21:29.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:21:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2402 2024-09-18T00:21:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2403 2024-09-18T00:21:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:21:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:21:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:21:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:21:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:21:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:21:31.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2405 2024-09-18T00:21:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2404 2024-09-18T00:21:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:21:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:21:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:21:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:21:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:21:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:21:31.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:21:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:21:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:21:32.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:21:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2407 2024-09-18T00:21:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2406 2024-09-18T00:21:32.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:33.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2409 2024-09-18T00:21:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2408 2024-09-18T00:21:33.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:21:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:21:33.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:21:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:21:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:33.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:21:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:21:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2411 2024-09-18T00:21:34.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2410 2024-09-18T00:21:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:21:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:21:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:21:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:21:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:21:34.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:21:35.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2413 2024-09-18T00:21:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:21:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:21:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:21:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:21:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:21:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:21:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2412 2024-09-18T00:21:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2414 2024-09-18T00:21:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2415 2024-09-18T00:21:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:21:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:21:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:21:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:21:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:36.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:21:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:21:36.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:36.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:21:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:21:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:21:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:21:37.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:37.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:37.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:21:37.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:21:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2416 2024-09-18T00:21:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2417 2024-09-18T00:21:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2418 2024-09-18T00:21:38.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2419 2024-09-18T00:21:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:21:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:21:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:21:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:38.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2421 2024-09-18T00:21:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2420 2024-09-18T00:21:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:21:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:21:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:21:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:21:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:21:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:21:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:21:39.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:21:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2422 2024-09-18T00:21:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2423 2024-09-18T00:21:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:21:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:21:40.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:21:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:21:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:21:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:21:41.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2424 2024-09-18T00:21:41.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2425 2024-09-18T00:21:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:41.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:21:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:21:41.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:21:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:21:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:21:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:21:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2426 2024-09-18T00:21:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:21:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:42.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:21:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:21:42.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:42.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:21:42.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:42.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:42.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:21:42.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:21:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:21:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:21:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:21:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:21:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:21:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:21:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2429 2024-09-18T00:21:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2427 2024-09-18T00:21:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2428 2024-09-18T00:21:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:43.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2431 2024-09-18T00:21:44.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2430 2024-09-18T00:21:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:21:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:21:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:21:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:21:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:21:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:21:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2433 2024-09-18T00:21:45.736 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:21:45.734+0000 7f6acfc5f640 -1 reset not still connected to 0x555c0bc54ea0 2024-09-18T00:21:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2432 2024-09-18T00:21:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:21:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:21:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:21:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2435 2024-09-18T00:21:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2434 2024-09-18T00:21:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:21:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:21:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:21:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:21:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:21:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:21:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:21:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:21:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:21:47.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:21:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:21:47.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:21:47.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:21:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2439 2024-09-18T00:21:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:21:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:21:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:21:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:21:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:21:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:21:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2436 2024-09-18T00:21:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2438 2024-09-18T00:21:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2437 2024-09-18T00:21:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2440 2024-09-18T00:21:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2441 2024-09-18T00:21:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:21:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:21:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:21:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:21:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:21:49.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:21:50.729 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2442 2024-09-18T00:21:50.730 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2443 2024-09-18T00:21:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:21:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:21:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:21:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2444 2024-09-18T00:21:51.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2445 2024-09-18T00:21:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:51.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:21:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:21:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:21:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:21:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:21:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:21:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:21:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:21:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:21:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2446 2024-09-18T00:21:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:21:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:21:52.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:21:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:21:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:21:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:21:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2447 2024-09-18T00:21:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2448 2024-09-18T00:21:53.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2449 2024-09-18T00:21:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:21:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:21:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:21:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:21:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:21:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:21:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2451 2024-09-18T00:21:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2450 2024-09-18T00:21:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:21:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:21:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:21:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:21:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:21:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:21:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:21:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:21:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:21:55.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2452 2024-09-18T00:21:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:21:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:21:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:21:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2453 2024-09-18T00:21:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2455 2024-09-18T00:21:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2454 2024-09-18T00:21:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:21:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:21:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:21:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:21:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:21:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:21:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:56.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2457 2024-09-18T00:21:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:21:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:21:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:21:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:21:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:21:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:21:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2459 2024-09-18T00:21:58.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2456 2024-09-18T00:21:58.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2458 2024-09-18T00:21:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:21:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:21:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:21:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2460 2024-09-18T00:21:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:21:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:21:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:21:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:21:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:21:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:21:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:21:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:21:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:21:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:21:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:21:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:21:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:22:00.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2461 2024-09-18T00:22:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2462 2024-09-18T00:22:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:22:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:22:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:22:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:00.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2463 2024-09-18T00:22:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2465 2024-09-18T00:22:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2464 2024-09-18T00:22:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:22:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:22:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:22:01.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:22:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:22:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:22:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:22:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:22:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:22:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2466 2024-09-18T00:22:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:22:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:22:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:22:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:02.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2469 2024-09-18T00:22:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2468 2024-09-18T00:22:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:22:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:22:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:22:03.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:03.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:22:03.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:22:03.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:22:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:22:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:22:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:22:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2467 2024-09-18T00:22:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:22:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:22:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:22:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:22:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:22:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:22:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2471 2024-09-18T00:22:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2472 2024-09-18T00:22:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2473 2024-09-18T00:22:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:22:05.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:22:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:22:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2470 2024-09-18T00:22:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:22:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:22:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:22:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2474 2024-09-18T00:22:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2475 2024-09-18T00:22:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:22:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:22:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:22:06.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2476 2024-09-18T00:22:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:22:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:22:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:22:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2477 2024-09-18T00:22:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:22:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:22:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:22:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2478 2024-09-18T00:22:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:22:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:22:08.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:22:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:22:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:22:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:22:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2479 2024-09-18T00:22:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:22:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:22:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:22:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2482 2024-09-18T00:22:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:22:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:22:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:22:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:22:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:22:09.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:22:10.007 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:22:10.005+0000 7f6acfc5f640 -1 reset not still connected to 0x555c1e000680 2024-09-18T00:22:10.007 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:22:10.005+0000 7f6acfc5f640 -1 reset not still connected to 0x555c1ec4f520 2024-09-18T00:22:10.007 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:22:10.005+0000 7f6acfc5f640 -1 reset not still connected to 0x555c200a2a90 2024-09-18T00:22:10.007 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:22:10.005+0000 7f6acfc5f640 -1 reset not still connected to 0x555c33a59110 2024-09-18T00:22:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2480 2024-09-18T00:22:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:22:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:22:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:22:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2481 2024-09-18T00:22:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2483 2024-09-18T00:22:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:22:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:22:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:22:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:10.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:10.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2484 2024-09-18T00:22:11.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2485 2024-09-18T00:22:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:22:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:22:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:22:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:22:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:22:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:22:12.401 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:22:12.399+0000 7f0114255640 -1 reset not still connected to 0x557d0a065e10 2024-09-18T00:22:12.401 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:22:12.399+0000 7f0114255640 -1 reset not still connected to 0x557d0b465110 2024-09-18T00:22:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:22:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:22:12.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:22:12.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2486 2024-09-18T00:22:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2487 2024-09-18T00:22:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2489 2024-09-18T00:22:13.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2488 2024-09-18T00:22:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:22:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:22:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:22:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:22:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:22:13.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:22:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:22:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:13.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:22:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:22:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2490 2024-09-18T00:22:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:14.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2491 2024-09-18T00:22:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:22:14.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:22:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:22:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:22:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:22:14.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:22:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2493 2024-09-18T00:22:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2492 2024-09-18T00:22:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:22:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:22:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:22:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:22:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:22:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:22:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2495 2024-09-18T00:22:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2494 2024-09-18T00:22:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:22:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:22:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:22:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:22:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:22:16.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:22:16.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:22:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:22:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:22:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:22:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:22:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:22:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2497 2024-09-18T00:22:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2496 2024-09-18T00:22:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2499 2024-09-18T00:22:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2498 2024-09-18T00:22:18.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:22:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:22:18.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:22:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:22:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:22:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:22:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2501 2024-09-18T00:22:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2500 2024-09-18T00:22:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:19.824 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:22:19.822+0000 7f9bfab49640 -1 reset not still connected to 0x55cd34597ad0 2024-09-18T00:22:19.824 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:22:19.822+0000 7f9bfab49640 -1 reset not still connected to 0x55cd434ae0d0 2024-09-18T00:22:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:22:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:22:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:22:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:22:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:22:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:22:20.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2502 2024-09-18T00:22:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2503 2024-09-18T00:22:20.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:20.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:22:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:22:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:22:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:22:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:22:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:22:21.020 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:22:21.019+0000 7f0114255640 -1 reset not still connected to 0x557d141fe340 2024-09-18T00:22:21.022 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:22:21.019+0000 7f0114255640 -1 reset not still connected to 0x557d14d3ca90 2024-09-18T00:22:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2505 2024-09-18T00:22:21.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2504 2024-09-18T00:22:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:22:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:22:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:22:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:22:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:22:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:22:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:22:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:22:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:22:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:22:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:22:22.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:22:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2506 2024-09-18T00:22:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2507 2024-09-18T00:22:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2509 2024-09-18T00:22:23.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2508 2024-09-18T00:22:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:22:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:22:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:22:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:22:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:22:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:22:23.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2511 2024-09-18T00:22:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2510 2024-09-18T00:22:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:22:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:22:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:22:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:22:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:22:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:22:25.015 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:22:25.012+0000 7f0114255640 -1 reset not still connected to 0x557d0a065ee0 2024-09-18T00:22:25.016 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:22:25.013+0000 7f0114255640 -1 reset not still connected to 0x557d0b464750 2024-09-18T00:22:25.016 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:22:25.013+0000 7f0114255640 -1 reset not still connected to 0x557d141fe340 2024-09-18T00:22:25.016 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:22:25.013+0000 7f0114255640 -1 reset not still connected to 0x557d14d3ca90 2024-09-18T00:22:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2512 2024-09-18T00:22:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:22:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:22:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:22:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2515 2024-09-18T00:22:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2513 2024-09-18T00:22:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2514 2024-09-18T00:22:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:22:26.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:26.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:26.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:22:26.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:22:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:22:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:22:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:22:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:22:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:22:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:22:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:22:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:22:27.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:22:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:22:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:22:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:22:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2516 2024-09-18T00:22:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2517 2024-09-18T00:22:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2519 2024-09-18T00:22:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2518 2024-09-18T00:22:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:22:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:22:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:22:29.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2520 2024-09-18T00:22:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2521 2024-09-18T00:22:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:22:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:22:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:22:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:22:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:22:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:22:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:22:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:22:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:22:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:22:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:22:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:22:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:22:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:22:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:22:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2525 2024-09-18T00:22:30.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2523 2024-09-18T00:22:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2524 2024-09-18T00:22:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2522 2024-09-18T00:22:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:22:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:22:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:22:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:22:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:22:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:22:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:31.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:22:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:22:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:22:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:22:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:22:32.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:22:32.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2526 2024-09-18T00:22:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2527 2024-09-18T00:22:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2528 2024-09-18T00:22:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2529 2024-09-18T00:22:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:33.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:22:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:22:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:22:34.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2531 2024-09-18T00:22:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2530 2024-09-18T00:22:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:34.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:22:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:22:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:22:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:22:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:22:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:22:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:22:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:22:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:22:35.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2532 2024-09-18T00:22:35.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2535 2024-09-18T00:22:35.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:22:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:22:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:22:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:22:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:22:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:22:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2533 2024-09-18T00:22:36.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2534 2024-09-18T00:22:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:22:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:22:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:22:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:22:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:22:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:22:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2536 2024-09-18T00:22:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:22:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:22:37.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:22:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:22:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:22:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:22:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:38.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2539 2024-09-18T00:22:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2537 2024-09-18T00:22:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2538 2024-09-18T00:22:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:22:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:22:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:22:38.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:22:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:22:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:22:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2540 2024-09-18T00:22:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:22:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:22:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:22:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:22:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:22:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:22:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2541 2024-09-18T00:22:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2543 2024-09-18T00:22:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:22:40.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:22:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:40.828 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:22:40.824+0000 7f9bfab49640 -1 reset not still connected to 0x55cd34597ad0 2024-09-18T00:22:40.828 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:22:40.824+0000 7f9bfab49640 -1 reset not still connected to 0x55cd434ae0d0 2024-09-18T00:22:40.848 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:22:40.845+0000 7f9bfab49640 -1 reset not still connected to 0x55cd34597ad0 2024-09-18T00:22:41.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2542 2024-09-18T00:22:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:22:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:22:41.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:22:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2544 2024-09-18T00:22:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:22:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:22:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:22:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:22:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:22:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:22:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2545 2024-09-18T00:22:42.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2546 2024-09-18T00:22:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:22:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:22:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:22:42.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:42.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2548 2024-09-18T00:22:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2549 2024-09-18T00:22:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:22:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:22:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:22:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:22:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:22:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:22:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2547 2024-09-18T00:22:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2550 2024-09-18T00:22:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2552 2024-09-18T00:22:44.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:22:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:22:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:22:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:22:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:22:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:22:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2551 2024-09-18T00:22:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:22:45.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:22:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:22:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:22:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:22:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:22:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2553 2024-09-18T00:22:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:22:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:22:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:22:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2554 2024-09-18T00:22:46.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:22:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:22:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:22:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2556 2024-09-18T00:22:46.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:22:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:22:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:22:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:22:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:22:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:22:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2555 2024-09-18T00:22:47.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2557 2024-09-18T00:22:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:22:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:22:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:22:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2558 2024-09-18T00:22:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2559 2024-09-18T00:22:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:22:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:22:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:22:48.823 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:48.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:22:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:22:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:22:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:49.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2560 2024-09-18T00:22:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2561 2024-09-18T00:22:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:49.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:22:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:22:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:22:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:22:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:22:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:22:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2563 2024-09-18T00:22:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:22:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:22:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:22:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:22:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:22:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:22:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2565 2024-09-18T00:22:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2562 2024-09-18T00:22:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2564 2024-09-18T00:22:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:22:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:22:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:22:51.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:22:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:22:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:22:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:22:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:22:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:22:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:22:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:22:52.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:22:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2566 2024-09-18T00:22:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2567 2024-09-18T00:22:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:53.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2568 2024-09-18T00:22:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2569 2024-09-18T00:22:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:22:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:22:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:22:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:53.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:54.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2570 2024-09-18T00:22:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2571 2024-09-18T00:22:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:22:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:22:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:22:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:22:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:22:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:22:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:22:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:22:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:22:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2572 2024-09-18T00:22:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:22:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:22:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:22:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2574 2024-09-18T00:22:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:22:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:22:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:22:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2573 2024-09-18T00:22:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2575 2024-09-18T00:22:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:22:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:22:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:56.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:22:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:22:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:22:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:22:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:22:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:22:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:22:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:22:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:22:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:22:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2577 2024-09-18T00:22:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2579 2024-09-18T00:22:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2578 2024-09-18T00:22:58.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2576 2024-09-18T00:22:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:22:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:22:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:22:58.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:22:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:22:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:22:58.909 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:22:58.907+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1e976820 2024-09-18T00:22:58.909 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:22:58.907+0000 7f9bfab49640 -1 reset not still connected to 0x55cd2b33a9c0 2024-09-18T00:22:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2581 2024-09-18T00:22:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2580 2024-09-18T00:22:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:22:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:22:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:22:59.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:22:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:22:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:22:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:22:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:22:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:22:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:23:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2583 2024-09-18T00:23:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2582 2024-09-18T00:23:00.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:23:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:23:00.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:23:00.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2584 2024-09-18T00:23:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2585 2024-09-18T00:23:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:23:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:23:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:23:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:23:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:23:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:23:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:23:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:23:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:23:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:23:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:02.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:23:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:23:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:23:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:23:02.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:23:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2586 2024-09-18T00:23:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2589 2024-09-18T00:23:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2587 2024-09-18T00:23:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2588 2024-09-18T00:23:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:23:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:23:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:23:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:23:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:23:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:23:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:04.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2591 2024-09-18T00:23:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2590 2024-09-18T00:23:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:23:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:23:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:04.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:04.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:04.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:23:04.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:23:04.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:23:04.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:23:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2592 2024-09-18T00:23:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2593 2024-09-18T00:23:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:23:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:23:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:23:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:23:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:23:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:23:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2595 2024-09-18T00:23:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2594 2024-09-18T00:23:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:23:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:23:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:23:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:23:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:23:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:23:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:23:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:23:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:23:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:23:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:23:07.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:23:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2596 2024-09-18T00:23:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2597 2024-09-18T00:23:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2598 2024-09-18T00:23:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2599 2024-09-18T00:23:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:23:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:23:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:23:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:23:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:23:08.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:23:08.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2600 2024-09-18T00:23:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2601 2024-09-18T00:23:09.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:23:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:23:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:23:09.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:09.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:23:09.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:09.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:09.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:23:09.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:23:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2602 2024-09-18T00:23:10.753 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2603 2024-09-18T00:23:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:23:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:23:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:23:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:23:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:23:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:23:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2604 2024-09-18T00:23:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2605 2024-09-18T00:23:11.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:23:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:23:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:23:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:23:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:23:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:23:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:23:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:23:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:23:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:23:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:23:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:23:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2606 2024-09-18T00:23:12.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2607 2024-09-18T00:23:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:23:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:23:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:23:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:23:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:23:13.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:23:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2608 2024-09-18T00:23:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2609 2024-09-18T00:23:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:14.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2610 2024-09-18T00:23:14.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2611 2024-09-18T00:23:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:23:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:23:14.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:23:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:23:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:23:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:23:15.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2612 2024-09-18T00:23:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2613 2024-09-18T00:23:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:15.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:23:15.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:15.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:15.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:23:15.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:23:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:23:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:23:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:23:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2615 2024-09-18T00:23:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2614 2024-09-18T00:23:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:23:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:23:16.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:23:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:23:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:23:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:23:17.001 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:23:16.999+0000 7fc275b74640 -1 reset not still connected to 0x559cc70b0410 2024-09-18T00:23:17.001 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:23:16.999+0000 7fc275b74640 -1 reset not still connected to 0x559ccd1ba340 2024-09-18T00:23:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2616 2024-09-18T00:23:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2617 2024-09-18T00:23:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:23:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:23:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:23:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:23:17.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 118735 2024-09-18T00:23:18.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2618 2024-09-18T00:23:18.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2619 2024-09-18T00:23:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:23:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:23:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:23:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:23:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 118867 2024-09-18T00:23:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2620 2024-09-18T00:23:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:23:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 118999 2024-09-18T00:23:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:23:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:23:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:23:20.763 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2621 2024-09-18T00:23:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:23:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:23:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:23:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:23:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 119151 2024-09-18T00:23:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2622 2024-09-18T00:23:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:23:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:23:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:23:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:23:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:23:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:23:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:23:21.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 119303 2024-09-18T00:23:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2623 2024-09-18T00:23:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:23:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 119456 2024-09-18T00:23:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2624 2024-09-18T00:23:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:23:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 119608 2024-09-18T00:23:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:23:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:23:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 119762 2024-09-18T00:23:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:23:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 119935 2024-09-18T00:23:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:23:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:23:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 118604 2024-09-18T00:25:49.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:25:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:25:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:25:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 118736 2024-09-18T00:25:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:25:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:25:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:25:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 118868 2024-09-18T00:25:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:25:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:25:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:25:51.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 119000 2024-09-18T00:25:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:25:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:25:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:25:52.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 119152 2024-09-18T00:25:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:25:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:25:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:25:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 119304 2024-09-18T00:25:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:25:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:25:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:25:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 119457 2024-09-18T00:25:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:25:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:25:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:25:55.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 119609 2024-09-18T00:25:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:25:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:25:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:25:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 119763 2024-09-18T00:25:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:25:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:25:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:25:57.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 119936 2024-09-18T00:25:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:25:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:25:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:25:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:25:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:25:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:25:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 1 2024-09-18T00:25:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:25:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:25:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 2 2024-09-18T00:25:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd0 2024-09-18T00:25:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary1 2024-09-18T00:25:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary1 2024-09-18T00:25:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:25:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary1 2024-09-18T00:25:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd1 2024-09-18T00:25:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:25:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary2 2024-09-18T00:25:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary2 2024-09-18T00:25:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:25:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary2 2024-09-18T00:25:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:25:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 3 2024-09-18T00:25:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd2 2024-09-18T00:25:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:25:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary3 2024-09-18T00:25:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary3 2024-09-18T00:25:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:25:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary3 2024-09-18T00:25:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:25:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 4 2024-09-18T00:25:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd3 2024-09-18T00:25:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary4 2024-09-18T00:25:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary4 2024-09-18T00:25:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:25:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary4 2024-09-18T00:25:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:25:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:25:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 5 2024-09-18T00:25:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd4 2024-09-18T00:25:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary5 2024-09-18T00:25:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary5 2024-09-18T00:25:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:25:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary5 2024-09-18T00:25:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:25:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:25:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 6 2024-09-18T00:25:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd5 2024-09-18T00:25:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:25:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:25:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 7 2024-09-18T00:25:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary6 2024-09-18T00:25:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary6 2024-09-18T00:25:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:25:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary6 2024-09-18T00:25:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd6 2024-09-18T00:25:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary7 2024-09-18T00:25:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary7 2024-09-18T00:25:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:25:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:25:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:25:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary7 2024-09-18T00:25:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:25:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:25:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:25:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:25:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 126651 2024-09-18T00:25:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 10 2024-09-18T00:25:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd9 2024-09-18T00:25:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary10 2024-09-18T00:25:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary10 2024-09-18T00:25:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:25:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary10 2024-09-18T00:25:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 8 2024-09-18T00:25:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 9 2024-09-18T00:25:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd8 2024-09-18T00:25:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary9 2024-09-18T00:25:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary9 2024-09-18T00:25:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:25:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary9 2024-09-18T00:25:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd7 2024-09-18T00:25:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary8 2024-09-18T00:25:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary8 2024-09-18T00:25:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:25:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary8 2024-09-18T00:25:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-18T00:25:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:25:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-18T00:25:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:25:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-18T00:25:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:25:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-18T00:25:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:25:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-18T00:25:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:25:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-18T00:25:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:25:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-18T00:25:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:25:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-18T00:25:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:25:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:25:58.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-18T00:25:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-18T00:25:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:26:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=d59cc5ed0d2bb894fcfda3d653d1bd11 2024-09-18T00:26:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-18T00:26:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-18T00:26:28.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:26:28.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-18T00:26:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=e8f41a40e042c9153978d75aeee1f634 2024-09-18T00:26:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-18T00:26:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=f291c89a5f54ee2f0eed7a79332d6f50 2024-09-18T00:26:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-18T00:26:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-18T00:26:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:26:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-18T00:26:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-18T00:26:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:26:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-18T00:26:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=c58372bf71746be2106ee73eec1974ea 2024-09-18T00:26:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-18T00:26:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-18T00:26:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:26:29.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-18T00:26:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=daa2bd85cb966c9695a41381a527e803 2024-09-18T00:26:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-18T00:26:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-18T00:26:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:26:29.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-18T00:26:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=e9b144779a2f98a3f73db0a69750afe4 2024-09-18T00:26:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-18T00:26:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=86e190a519d17579bac8ff550357a15f 2024-09-18T00:26:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-18T00:26:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:26:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-18T00:26:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:26:29.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-18T00:26:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=f9ed88af1c6746f783a997ae041e9a15 2024-09-18T00:26:29.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-18T00:26:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-18T00:26:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:26:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-18T00:26:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:26:29.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-18T00:26:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=1c9a079c0791d816d6469e346c5a3ae6 2024-09-18T00:26:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-18T00:26:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-18T00:26:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:26:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-18T00:26:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-18T00:26:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:26:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-18T00:26:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=a1e6354e288e8f89713a3ed69c80abdb 2024-09-18T00:26:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-18T00:26:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-18T00:26:30.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:26:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:26:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:26:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:26:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:26:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:26:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-18T00:26:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:26:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-18T00:26:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:26:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-18T00:26:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-18T00:26:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:26:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-18T00:26:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:26:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:30.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-18T00:26:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:26:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:30.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:26:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 3a078d42-4f72-4a33-8636-ba1da94d8aa6 2024-09-18T00:26:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:26:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:26:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:27 2024-09-18T00:26:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:26:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopping_replay 2024-09-18T00:26:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:26:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:31' 2024-09-18T00:26:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:26:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:26:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:26:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:26:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:26:31.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:26:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-18T00:26:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:26:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-18T00:26:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:26:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-18T00:26:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:26:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:31.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-18T00:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-18T00:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:31.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:26:31.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:31.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:31.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:31.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:31.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-18T00:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:31.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:26:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-18T00:26:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:26:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:31.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:26:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-18T00:26:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:26:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:26:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:26:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1745ddfe-ba46-4c43-8c01-3b0b11ca89ce 2024-09-18T00:26:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:26:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:26:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:27 2024-09-18T00:26:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:26:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:26:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:31' 2024-09-18T00:26:32.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:26:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:26:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 10dacb73-f415-417d-8ed1-6bdb0fc5fcf2 2024-09-18T00:26:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:26:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:26:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:27 2024-09-18T00:26:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:26:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:26:32.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:26:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:31' 2024-09-18T00:26:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:26:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:32.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:26:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:26:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9ee95762-996e-4381-9c29-8aeddf45ccc6 2024-09-18T00:26:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:26:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:26:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:27 2024-09-18T00:26:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:26:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:26:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:31' 2024-09-18T00:26:32.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:26:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:26:32.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:26:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1af2badd-a447-4ef2-be7f-7f2e9a23b3c7 2024-09-18T00:26:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:26:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:26:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:27 2024-09-18T00:26:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:26:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:26:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:31' 2024-09-18T00:26:32.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:26:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:26:32.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:26:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-18T00:26:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:26:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-18T00:26:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:26:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:32.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:26:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-18T00:26:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:26:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:32.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:32.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-18T00:26:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-18T00:26:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:26:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:32.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-18T00:26:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-18T00:26:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:26:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:32.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:26:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:26:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d08650b4-8cb9-4025-8199-1719a7f72860 2024-09-18T00:26:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:26:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:26:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3590144.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619001,"remote_snapshot_timestamp":1726619001,"replay_state":"idle"} 2024-09-18T00:26:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:17' 2024-09-18T00:26:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:33.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:33.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:33.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:26:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ce11166a-ff96-4612-8765-db39b0dba9f3 2024-09-18T00:26:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:33.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:26:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:26:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:33.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:26:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:26:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: aa5f09f2-9a9e-4dac-b2c1-219170bfb686 2024-09-18T00:26:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:26:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:26:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c5dfa332-f995-41a3-a007-b88c3f9ab788 2024-09-18T00:26:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:26:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-18T00:26:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:26:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:33.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:26:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:26:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:26:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:33.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33' 2024-09-18T00:26:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:33.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:26:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:26:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:33.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-18T00:26:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:26:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:33.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:26:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 3a078d42-4f72-4a33-8636-ba1da94d8aa6 2024-09-18T00:26:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:33.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:26:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:26:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33' 2024-09-18T00:26:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:26:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:26:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-18T00:26:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:26:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33' 2024-09-18T00:26:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:33.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33' 2024-09-18T00:26:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:26:33.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:26:33.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:26:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:26:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:33.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-18T00:26:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:26:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-18T00:26:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:26:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33' 2024-09-18T00:26:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:26:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:26:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-18T00:26:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:26:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-18T00:26:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:26:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:26:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:26:34.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8ae5cefc-b421-4d99-a945-ce61f7358a1b 2024-09-18T00:26:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:26:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:26:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:33' 2024-09-18T00:26:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:34.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:26:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-18T00:26:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:26:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:26:34.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-18T00:26:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:26:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:26:34.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-18T00:26:34.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:34.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:26:34.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:34.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:26:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-18T00:26:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:26:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:26:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33' 2024-09-18T00:26:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:26:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:26:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-18T00:26:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:26:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:26:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1745ddfe-ba46-4c43-8c01-3b0b11ca89ce 2024-09-18T00:26:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:26:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:34.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:26:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9ee95762-996e-4381-9c29-8aeddf45ccc6 2024-09-18T00:26:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:34.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:34.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:34.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:26:34.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:34.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:34.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:34.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:26:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:26:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1af2badd-a447-4ef2-be7f-7f2e9a23b3c7 2024-09-18T00:26:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:34.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:34.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:34.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:26:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-18T00:26:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:26:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-18T00:26:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-18T00:26:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:26:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-18T00:26:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:26:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d08650b4-8cb9-4025-8199-1719a7f72860 2024-09-18T00:26:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:26:34.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:33 2024-09-18T00:26:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:26:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:34.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:34.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:26:34.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:33' 2024-09-18T00:26:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-18 00:26:33' 2024-09-18T00:26:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:26:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:33' 2024-09-18T00:26:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:34.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-18T00:26:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-18T00:26:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:26:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:34.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:26:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-18T00:26:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-18T00:26:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33' 2024-09-18T00:26:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:34.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:26:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:26:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-18T00:26:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:26:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-18T00:26:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:26:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-18T00:26:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33' 2024-09-18T00:26:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:26:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:26:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33' 2024-09-18T00:26:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:26:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-18T00:26:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:26:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:34.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:26:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:26:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: aa5f09f2-9a9e-4dac-b2c1-219170bfb686 2024-09-18T00:26:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:26:34.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-18T00:26:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:26:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:26:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c5dfa332-f995-41a3-a007-b88c3f9ab788 2024-09-18T00:26:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:26:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:34.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:34.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:34.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:26:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:26:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32' 2024-09-18T00:26:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:34.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:26:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:26:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 10dacb73-f415-417d-8ed1-6bdb0fc5fcf2 2024-09-18T00:26:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:26:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:33' 2024-09-18T00:26:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:26:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-18T00:26:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:26:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:26:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-18T00:26:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:34.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:26:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:26:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:26:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 3a078d42-4f72-4a33-8636-ba1da94d8aa6 2024-09-18T00:26:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:26:34.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:34.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:34.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:26:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32' 2024-09-18T00:26:34.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:26:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:26:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32' 2024-09-18T00:26:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:26:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-18T00:26:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:26:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:26:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-18T00:26:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:26:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ce11166a-ff96-4612-8765-db39b0dba9f3 2024-09-18T00:26:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:26:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:26:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:26:32cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:26:33' 2024-09-18T00:26:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:26:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.517 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-18T00:26:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:26:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:26:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-18T00:26:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:26:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:26:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-18T00:26:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:26:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:26:34.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:26:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32' 2024-09-18T00:26:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:26:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.558 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-18T00:26:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:26:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:26:34.560 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-18T00:26:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32' 2024-09-18T00:26:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:26:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:26:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:34.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:34.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-18T00:26:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:26:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:26:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-18T00:26:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:26:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:26:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:26:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:26:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:26:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:26:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:26:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:26:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:26:35.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:26:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:26:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-18T00:26:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-18T00:26:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-18T00:26:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:26:35.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:26:35.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-18T00:26:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:26:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:26:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-18T00:26:35.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-18T00:26:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:26:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-18T00:26:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd0 2024-09-18T00:26:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-18T00:26:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:26:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd1 2024-09-18T00:26:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-18T00:26:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd2 2024-09-18T00:26:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:26:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-18T00:26:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:26:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd3 2024-09-18T00:26:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd4 2024-09-18T00:26:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-18T00:26:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-18T00:26:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:26:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:26:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:26:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8ae5cefc-b421-4d99-a945-ce61f7358a1b 2024-09-18T00:26:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:26:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:33 2024-09-18T00:26:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:26:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:26:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:33' 2024-09-18T00:26:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-18 00:26:33' 2024-09-18T00:26:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:26:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:33' 2024-09-18T00:26:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-18T00:26:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:26:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-18T00:26:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-18T00:26:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:26:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-18T00:26:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-18T00:26:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:26:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-18T00:26:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:26:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1745ddfe-ba46-4c43-8c01-3b0b11ca89ce 2024-09-18T00:26:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:26:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:35.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:35.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:26:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9ee95762-996e-4381-9c29-8aeddf45ccc6 2024-09-18T00:26:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:26:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32' 2024-09-18T00:26:35.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:26:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:35.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:35.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:35.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:35.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:33' 2024-09-18T00:26:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:26:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:26:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1af2badd-a447-4ef2-be7f-7f2e9a23b3c7 2024-09-18T00:26:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:26:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:35.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:26:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:35.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-18T00:26:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:26:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:26:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-18T00:26:35.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:35.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:26:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:35.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:26:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-18T00:26:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:26:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:26:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:26:35.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:35.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-18T00:26:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32' 2024-09-18T00:26:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:35.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:26:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:35.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:26:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-18T00:26:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:26:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:26:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-18T00:26:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32' 2024-09-18T00:26:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32' 2024-09-18T00:26:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:26:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:26:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:26:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:26:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:26:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 10dacb73-f415-417d-8ed1-6bdb0fc5fcf2 2024-09-18T00:26:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:26:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32 2024-09-18T00:26:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:26:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:26:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:26:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:26:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:32' 2024-09-18T00:26:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:26:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:26:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:26:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:26:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:26:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-18T00:26:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-18T00:26:35.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:26:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:26:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-18T00:26:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:26:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:26:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:26:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-18T00:26:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:26:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:26:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:26:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:26:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-18 00:26:32' 2024-09-18T00:26:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:26:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:26:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:32' 2024-09-18T00:26:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:26:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:26:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:26:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:26:32cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:26:33cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:26:32' 2024-09-18T00:26:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:26:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:26:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:32' 2024-09-18T00:26:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:26:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:26:35.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:26:35.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:26:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:26:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:26:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-18T00:26:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:26:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:26:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:26:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:26:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-18T00:26:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:26:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:26:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:26:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:26:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:26:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:26:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-18T00:26:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:26:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:26:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:26:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:26:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-18T00:26:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:26:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:26:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-18T00:26:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-18T00:26:36.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd6 2024-09-18T00:26:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd5 2024-09-18T00:26:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-18T00:26:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:26:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-18T00:26:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:26:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd7 2024-09-18T00:26:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-18T00:26:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:26:36.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd8 2024-09-18T00:26:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-18T00:26:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:26:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:26:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:26:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:26:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-18T00:26:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd9 2024-09-18T00:26:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-18T00:26:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:27:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=86e190a519d17579bac8ff550357a15f 2024-09-18T00:27:05.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-18T00:27:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 86e190a519d17579bac8ff550357a15f != \8\6\e\1\9\0\a\5\1\9\d\1\7\5\7\9\b\a\c\8\f\f\5\5\0\3\5\7\a\1\5\f ]] 2024-09-18T00:27:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=d59cc5ed0d2bb894fcfda3d653d1bd11 2024-09-18T00:27:06.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-18T00:27:06.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ d59cc5ed0d2bb894fcfda3d653d1bd11 != \d\5\9\c\c\5\e\d\0\d\2\b\b\8\9\4\f\c\f\d\a\3\d\6\5\3\d\1\b\d\1\1 ]] 2024-09-18T00:27:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=1c9a079c0791d816d6469e346c5a3ae6 2024-09-18T00:27:06.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-18T00:27:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 1c9a079c0791d816d6469e346c5a3ae6 != \1\c\9\a\0\7\9\c\0\7\9\1\d\8\1\6\d\6\4\6\9\e\3\4\6\c\5\a\3\a\e\6 ]] 2024-09-18T00:27:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=f291c89a5f54ee2f0eed7a79332d6f50 2024-09-18T00:27:07.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-18T00:27:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ f291c89a5f54ee2f0eed7a79332d6f50 != \f\2\9\1\c\8\9\a\5\f\5\4\e\e\2\f\0\e\e\d\7\a\7\9\3\3\2\d\6\f\5\0 ]] 2024-09-18T00:27:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=f9ed88af1c6746f783a997ae041e9a15 2024-09-18T00:27:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-18T00:27:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ f9ed88af1c6746f783a997ae041e9a15 != \f\9\e\d\8\8\a\f\1\c\6\7\4\6\f\7\8\3\a\9\9\7\a\e\0\4\1\e\9\a\1\5 ]] 2024-09-18T00:27:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=c58372bf71746be2106ee73eec1974ea 2024-09-18T00:27:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-18T00:27:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ c58372bf71746be2106ee73eec1974ea != \c\5\8\3\7\2\b\f\7\1\7\4\6\b\e\2\1\0\6\e\e\7\3\e\e\c\1\9\7\4\e\a ]] 2024-09-18T00:27:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=daa2bd85cb966c9695a41381a527e803 2024-09-18T00:27:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-18T00:27:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ daa2bd85cb966c9695a41381a527e803 != \d\a\a\2\b\d\8\5\c\b\9\6\6\c\9\6\9\5\a\4\1\3\8\1\a\5\2\7\e\8\0\3 ]] 2024-09-18T00:27:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=e9b144779a2f98a3f73db0a69750afe4 2024-09-18T00:27:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-18T00:27:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=e8f41a40e042c9153978d75aeee1f634 2024-09-18T00:27:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-18T00:27:08.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ e9b144779a2f98a3f73db0a69750afe4 != \e\9\b\1\4\4\7\7\9\a\2\f\9\8\a\3\f\7\3\d\b\0\a\6\9\7\5\0\a\f\e\4 ]] 2024-09-18T00:27:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ e8f41a40e042c9153978d75aeee1f634 != \e\8\f\4\1\a\4\0\e\0\4\2\c\9\1\5\3\9\7\8\d\7\5\a\e\e\e\1\f\6\3\4 ]] 2024-09-18T00:27:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=a1e6354e288e8f89713a3ed69c80abdb 2024-09-18T00:27:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-18T00:27:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ a1e6354e288e8f89713a3ed69c80abdb != \a\1\e\6\3\5\4\e\2\8\8\e\8\f\8\9\7\1\3\a\3\e\d\6\9\c\8\0\a\b\d\b ]] 2024-09-18T00:27:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:27:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 126652 2024-09-18T00:27:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:27:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 126653 2024-09-18T00:27:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:27:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 126656 2024-09-18T00:27:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:27:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 126658 2024-09-18T00:27:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:27:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 126660 2024-09-18T00:27:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:27:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 126662 2024-09-18T00:27:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 126664 2024-09-18T00:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 126665 2024-09-18T00:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 126667 2024-09-18T00:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-18T00:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-18T00:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:27:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-18T00:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:27:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-18T00:27:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:27:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:27:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:27:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 8ae5cefc-b421-4d99-a945-ce61f7358a1b 2024-09-18T00:27:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:27:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":7438336.0,"bytes_per_snapshot":61415424.0,"last_snapshot_bytes":61415424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"} 2024-09-18T00:27:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:27:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:47 2024-09-18T00:27:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:27:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:27:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:27:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:27:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:07' 2024-09-18T00:27:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:27:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:27:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:27:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":7438336.0,"bytes_per_snapshot":61415424.0,"last_snapshot_bytes":61415424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"} 2024-09-18T00:27:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:27:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:27:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:27:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:27:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:27:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:27:10.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:47' 2024-09-18T00:27:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-18 00:26:47' 2024-09-18T00:27:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:27:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:47' 2024-09-18T00:27:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:27:10.359 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:27:10.357+0000 7f9bfab49640 -1 reset not still connected to 0x55cd282c6c30 2024-09-18T00:27:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary18ae5cefc-b421-4d99-a945-ce61f7358a1bup+replayingreplaying, {"bytes_per_second":7438336.0,"bytes_per_snapshot":61415424.0,"last_snapshot_bytes":61415424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:07image-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+replayingreplaying, {"bytes_per_second":30765056.0,"bytes_per_snapshot":61530112.0,"last_snapshot_bytes":61530112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619197,"remote_snapshot_timestamp":1726619197,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:07image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+replayingreplaying, {"bytes_per_second":13246464.0,"bytes_per_snapshot":64749568.0,"last_snapshot_bytes":64749568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:07image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61460480.0,"last_snapshot_bytes":61460480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:07image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+replayingreplaying, {"bytes_per_second":31526912.0,"bytes_per_snapshot":63053824.0,"last_snapshot_bytes":63053824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:07image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64598016.0,"last_snapshot_bytes":64598016,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:07image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61599744.0,"last_snapshot_bytes":61599744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:07image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63025152.0,"last_snapshot_bytes":63025152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:07image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+replayingreplaying, {"bytes_per_second":30810112.0,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:07image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63074304.0,"last_snapshot_bytes":63074304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:07' 2024-09-18T00:27:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:27:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:27:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:47' 2024-09-18T00:27:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:27:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:27:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:10.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:27:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-18T00:27:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:27:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:27:10.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-18T00:27:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-18T00:27:12.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:27:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-18T00:27:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:27:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:27:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary1 2024-09-18T00:27:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary1 2024-09-18T00:27:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-18T00:27:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-18T00:27:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-18T00:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-18T00:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:27:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:27:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 3a078d42-4f72-4a33-8636-ba1da94d8aa6 2024-09-18T00:27:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:27:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":13246464.0,"bytes_per_snapshot":64749568.0,"last_snapshot_bytes":64749568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"} 2024-09-18T00:27:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:27:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:47 2024-09-18T00:27:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:27:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:12' 2024-09-18T00:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:27:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:27:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":13246464.0,"bytes_per_snapshot":64749568.0,"last_snapshot_bytes":64749568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"} 2024-09-18T00:27:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:27:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:27:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:27:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:27:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:27:13.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:27:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:47' 2024-09-18T00:27:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-18 00:26:47' 2024-09-18T00:27:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:27:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:47' 2024-09-18T00:27:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:13.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:27:13.346 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:27:13.344+0000 7f6acfc5f640 -1 reset not still connected to 0x555c0e1dc680 2024-09-18T00:27:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK942634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+replayingreplaying, {"bytes_per_second":30765056.0,"bytes_per_snapshot":61530112.0,"last_snapshot_bytes":61530112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619197,"remote_snapshot_timestamp":1726619197,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12image-primary23a078d42-4f72-4a33-8636-ba1da94d8aa6up+replayingreplaying, {"bytes_per_second":13246464.0,"bytes_per_snapshot":64749568.0,"last_snapshot_bytes":64749568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61460480.0,"last_snapshot_bytes":61460480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+replayingreplaying, {"bytes_per_second":31526912.0,"bytes_per_snapshot":63053824.0,"last_snapshot_bytes":63053824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64598016.0,"last_snapshot_bytes":64598016,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61599744.0,"last_snapshot_bytes":61599744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63025152.0,"last_snapshot_bytes":63025152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+replayingreplaying, {"bytes_per_second":30810112.0,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63074304.0,"last_snapshot_bytes":63074304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12' 2024-09-18T00:27:13.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:27:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:27:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:47' 2024-09-18T00:27:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:27:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:27:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:27:13.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-18T00:27:13.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:27:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:27:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-18T00:27:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-18T00:27:15.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:27:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-18T00:27:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:27:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:27:15.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary2 2024-09-18T00:27:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary2 2024-09-18T00:27:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-18T00:27:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-18T00:27:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:27:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:27:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-18T00:27:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:27:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:27:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-18T00:27:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:27:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:27:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:27:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1745ddfe-ba46-4c43-8c01-3b0b11ca89ce 2024-09-18T00:27:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:27:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61460480.0,"last_snapshot_bytes":61460480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"} 2024-09-18T00:27:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:27:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:26:47 2024-09-18T00:27:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:27:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:27:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:27:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:27:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:12' 2024-09-18T00:27:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:27:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:27:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:27:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61460480.0,"last_snapshot_bytes":61460480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"} 2024-09-18T00:27:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:27:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:27:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:27:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:27:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:27:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:27:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:47' 2024-09-18T00:27:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-18 00:26:47' 2024-09-18T00:27:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:27:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:26:47' 2024-09-18T00:27:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:16.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:27:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK842634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+replayingreplaying, {"bytes_per_second":30765056.0,"bytes_per_snapshot":61530112.0,"last_snapshot_bytes":61530112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619197,"remote_snapshot_timestamp":1726619197,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12image-primary31745ddfe-ba46-4c43-8c01-3b0b11ca89ceup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61460480.0,"last_snapshot_bytes":61460480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+replayingreplaying, {"bytes_per_second":31526912.0,"bytes_per_snapshot":63053824.0,"last_snapshot_bytes":63053824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64598016.0,"last_snapshot_bytes":64598016,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61599744.0,"last_snapshot_bytes":61599744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63025152.0,"last_snapshot_bytes":63025152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+replayingreplaying, {"bytes_per_second":30810112.0,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63074304.0,"last_snapshot_bytes":63074304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:26:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:12' 2024-09-18T00:27:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:27:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:27:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:26:47' 2024-09-18T00:27:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:27:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:27:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:16.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:27:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-18T00:27:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:27:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:27:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-18T00:27:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-18T00:27:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:27:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-18T00:27:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:27:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:27:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary3 2024-09-18T00:27:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary3 2024-09-18T00:27:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-18T00:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-18T00:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-18T00:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:27:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-18T00:27:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:19.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:27:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:19.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-18T00:27:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:27:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ce11166a-ff96-4612-8765-db39b0dba9f3 2024-09-18T00:27:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:27:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2101794.13,"bytes_per_snapshot":63053824.0,"last_snapshot_bytes":63053824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"} 2024-09-18T00:27:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:27:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:17 2024-09-18T00:27:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:27:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:27:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:27:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:27:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:17' 2024-09-18T00:27:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:27:19.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:27:19.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:27:19.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2101794.13,"bytes_per_snapshot":63053824.0,"last_snapshot_bytes":63053824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"} 2024-09-18T00:27:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:27:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:27:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:27:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:27:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:27:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:27:19.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:27:17' 2024-09-18T00:27:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-18 00:27:17' 2024-09-18T00:27:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:27:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:27:17' 2024-09-18T00:27:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:27:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK742634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+replayingreplaying, {"bytes_per_second":2051003.73,"bytes_per_snapshot":61530112.0,"last_snapshot_bytes":61530112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619197,"remote_snapshot_timestamp":1726619197,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:17image-primary4ce11166a-ff96-4612-8765-db39b0dba9f3up+replayingreplaying, {"bytes_per_second":2101794.13,"bytes_per_snapshot":63053824.0,"last_snapshot_bytes":63053824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:17image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+replayingreplaying, {"bytes_per_second":2153267.2,"bytes_per_snapshot":64598016.0,"last_snapshot_bytes":64598016,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:17image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+replayingreplaying, {"bytes_per_second":2053324.8,"bytes_per_snapshot":61599744.0,"last_snapshot_bytes":61599744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:17image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+replayingreplaying, {"bytes_per_second":2100838.4,"bytes_per_snapshot":63025152.0,"last_snapshot_bytes":63025152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:17image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+replayingreplaying, {"bytes_per_second":2109576.53,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:17image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+replayingreplaying, {"bytes_per_second":2102476.8,"bytes_per_snapshot":63074304.0,"last_snapshot_bytes":63074304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:17' 2024-09-18T00:27:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:27:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:27:19.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:27:17' 2024-09-18T00:27:19.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:27:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:27:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:27:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-18T00:27:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:27:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:27:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-18T00:27:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-18T00:27:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:27:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-18T00:27:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:27:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:27:21.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary4 2024-09-18T00:27:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary4 2024-09-18T00:27:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-18T00:27:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-18T00:27:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:27:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:27:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-18T00:27:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:27:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:27:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-18T00:27:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:27:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-18T00:27:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:27:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: aa5f09f2-9a9e-4dac-b2c1-219170bfb686 2024-09-18T00:27:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:27:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2153267.2,"bytes_per_snapshot":64598016.0,"last_snapshot_bytes":64598016,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"} 2024-09-18T00:27:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:27:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:17 2024-09-18T00:27:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:27:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:27:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:27:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:27:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:17' 2024-09-18T00:27:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:27:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:27:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:27:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2153267.2,"bytes_per_snapshot":64598016.0,"last_snapshot_bytes":64598016,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"} 2024-09-18T00:27:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:27:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:27:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:27:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:27:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:27:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:27:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:27:17' 2024-09-18T00:27:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-18 00:27:17' 2024-09-18T00:27:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:27:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:27:17' 2024-09-18T00:27:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:27:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK642634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+replayingreplaying, {"bytes_per_second":2051003.73,"bytes_per_snapshot":61530112.0,"last_snapshot_bytes":61530112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619197,"remote_snapshot_timestamp":1726619197,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:17image-primary5aa5f09f2-9a9e-4dac-b2c1-219170bfb686up+replayingreplaying, {"bytes_per_second":2153267.2,"bytes_per_snapshot":64598016.0,"last_snapshot_bytes":64598016,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:17image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+replayingreplaying, {"bytes_per_second":2053324.8,"bytes_per_snapshot":61599744.0,"last_snapshot_bytes":61599744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:17image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+replayingreplaying, {"bytes_per_second":2100838.4,"bytes_per_snapshot":63025152.0,"last_snapshot_bytes":63025152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:17image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+replayingreplaying, {"bytes_per_second":2109576.53,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:17image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+replayingreplaying, {"bytes_per_second":2102476.8,"bytes_per_snapshot":63074304.0,"last_snapshot_bytes":63074304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:17' 2024-09-18T00:27:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:27:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:27:22.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:27:17' 2024-09-18T00:27:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:27:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:27:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:27:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-18T00:27:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:27:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:27:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-18T00:27:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-18T00:27:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:27:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-18T00:27:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:27:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:27:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary5 2024-09-18T00:27:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary5 2024-09-18T00:27:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-18T00:27:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-18T00:27:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:27:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:27:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-18T00:27:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:27:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:27:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-18T00:27:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:27:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:27:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:27:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1af2badd-a447-4ef2-be7f-7f2e9a23b3c7 2024-09-18T00:27:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:27:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2053324.8,"bytes_per_snapshot":61599744.0,"last_snapshot_bytes":61599744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"} 2024-09-18T00:27:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:27:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:17 2024-09-18T00:27:25.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:27:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:27:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:27:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:27:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:22' 2024-09-18T00:27:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:27:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:27:25.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:27:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2053324.8,"bytes_per_snapshot":61599744.0,"last_snapshot_bytes":61599744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"} 2024-09-18T00:27:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:27:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:25.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:27:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:27:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:27:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:27:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:27:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:27:17' 2024-09-18T00:27:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-18 00:27:17' 2024-09-18T00:27:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:27:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:27:17' 2024-09-18T00:27:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:27:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK542634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+replayingreplaying, {"bytes_per_second":2051003.73,"bytes_per_snapshot":61530112.0,"last_snapshot_bytes":61530112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619197,"remote_snapshot_timestamp":1726619197,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:22image-primary61af2badd-a447-4ef2-be7f-7f2e9a23b3c7up+replayingreplaying, {"bytes_per_second":2053324.8,"bytes_per_snapshot":61599744.0,"last_snapshot_bytes":61599744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:22image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+replayingreplaying, {"bytes_per_second":2100838.4,"bytes_per_snapshot":63025152.0,"last_snapshot_bytes":63025152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:22image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+replayingreplaying, {"bytes_per_second":2109576.53,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:22image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+replayingreplaying, {"bytes_per_second":2102476.8,"bytes_per_snapshot":63074304.0,"last_snapshot_bytes":63074304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:22' 2024-09-18T00:27:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:27:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:27:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:27:17' 2024-09-18T00:27:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:27:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:27:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:27:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-18T00:27:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:27:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:27:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-18T00:27:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-18T00:27:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:27:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-18T00:27:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:27:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:27:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary6 2024-09-18T00:27:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary6 2024-09-18T00:27:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-18T00:27:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-18T00:27:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:27:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:27:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-18T00:27:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:27:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:27:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-18T00:27:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:27:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:28.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:27:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:27:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9ee95762-996e-4381-9c29-8aeddf45ccc6 2024-09-18T00:27:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:27:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2100838.4,"bytes_per_snapshot":63025152.0,"last_snapshot_bytes":63025152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"} 2024-09-18T00:27:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:27:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:17 2024-09-18T00:27:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:27:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:27:28.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:27:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:27:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:27' 2024-09-18T00:27:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:27:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:27:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:27:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2100838.4,"bytes_per_snapshot":63025152.0,"last_snapshot_bytes":63025152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"} 2024-09-18T00:27:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:27:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:28.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:27:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:28.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:27:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:27:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:27:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:27:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:27:17' 2024-09-18T00:27:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-18 00:27:17' 2024-09-18T00:27:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:27:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:27:17' 2024-09-18T00:27:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:27:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+replayingreplaying, {"bytes_per_second":2051003.73,"bytes_per_snapshot":61530112.0,"last_snapshot_bytes":61530112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619197,"remote_snapshot_timestamp":1726619197,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:27image-primary79ee95762-996e-4381-9c29-8aeddf45ccc6up+replayingreplaying, {"bytes_per_second":2100838.4,"bytes_per_snapshot":63025152.0,"last_snapshot_bytes":63025152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619196,"remote_snapshot_timestamp":1726619196,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:27image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+replayingreplaying, {"bytes_per_second":2109576.53,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:27image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+replayingreplaying, {"bytes_per_second":2102476.8,"bytes_per_snapshot":63074304.0,"last_snapshot_bytes":63074304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:27' 2024-09-18T00:27:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:27:28.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:27:28.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:27:17' 2024-09-18T00:27:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:27:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:27:28.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:27:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-18T00:27:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:27:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:27:28.375 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-18T00:27:28.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-18T00:27:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:27:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-18T00:27:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:27:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:27:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary7 2024-09-18T00:27:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary7 2024-09-18T00:27:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-18T00:27:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-18T00:27:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:27:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:27:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-18T00:27:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:27:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:27:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-18T00:27:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:27:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:27:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:27:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c5dfa332-f995-41a3-a007-b88c3f9ab788 2024-09-18T00:27:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:27:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2109576.53,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"} 2024-09-18T00:27:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:27:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:17 2024-09-18T00:27:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:27:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:27:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:27:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:27:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:27' 2024-09-18T00:27:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:27:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:27:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:27:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2109576.53,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"} 2024-09-18T00:27:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:27:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:27:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:31.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:27:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:27:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:27:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:27:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:27:17' 2024-09-18T00:27:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-18 00:27:17' 2024-09-18T00:27:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:27:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:27:17' 2024-09-18T00:27:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:27:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK342634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+replayingreplaying, {"bytes_per_second":2051003.73,"bytes_per_snapshot":61530112.0,"last_snapshot_bytes":61530112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619197,"remote_snapshot_timestamp":1726619197,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:27image-primary8c5dfa332-f995-41a3-a007-b88c3f9ab788up+replayingreplaying, {"bytes_per_second":2109576.53,"bytes_per_snapshot":63287296.0,"last_snapshot_bytes":63287296,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:27image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+replayingreplaying, {"bytes_per_second":2102476.8,"bytes_per_snapshot":63074304.0,"last_snapshot_bytes":63074304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:27' 2024-09-18T00:27:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:27:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:27:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:27:17' 2024-09-18T00:27:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:27:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:27:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:27:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-18T00:27:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:27:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:27:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-18T00:27:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-18T00:27:33.180 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:27:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-18T00:27:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:27:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:27:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary8 2024-09-18T00:27:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:33.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary8 2024-09-18T00:27:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-18T00:27:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-18T00:27:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:27:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:27:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-18T00:27:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:27:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:27:33.329 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:27:33.325+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4dd0 2024-09-18T00:27:33.329 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:27:33.326+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb5790 2024-09-18T00:27:33.329 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:27:33.326+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ea270 2024-09-18T00:27:33.329 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:27:33.326+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:27:33.329 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:27:33.326+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-18T00:27:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-18T00:27:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:27:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:27:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:27:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: d08650b4-8cb9-4025-8199-1719a7f72860 2024-09-18T00:27:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:27:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2102476.8,"bytes_per_snapshot":63074304.0,"last_snapshot_bytes":63074304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"} 2024-09-18T00:27:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:27:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:17 2024-09-18T00:27:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:27:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:27:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:27:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:27:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:32' 2024-09-18T00:27:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:27:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:27:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:27:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2102476.8,"bytes_per_snapshot":63074304.0,"last_snapshot_bytes":63074304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"} 2024-09-18T00:27:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:27:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:27:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:27:34.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:27:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:27:34.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:27:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:27:17' 2024-09-18T00:27:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-18 00:27:17' 2024-09-18T00:27:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:27:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:27:17' 2024-09-18T00:27:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:34.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:27:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK242634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+replayingreplaying, {"bytes_per_second":2051003.73,"bytes_per_snapshot":61530112.0,"last_snapshot_bytes":61530112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619197,"remote_snapshot_timestamp":1726619197,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:32image-primary9d08650b4-8cb9-4025-8199-1719a7f72860up+replayingreplaying, {"bytes_per_second":2102476.8,"bytes_per_snapshot":63074304.0,"last_snapshot_bytes":63074304,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619195,"remote_snapshot_timestamp":1726619195,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:32' 2024-09-18T00:27:34.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:27:34.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:27:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:27:17' 2024-09-18T00:27:34.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:27:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:27:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:27:34.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-18T00:27:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:27:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:27:34.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-18T00:27:34.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-18T00:27:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:27:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-18T00:27:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:27:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:27:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary9 2024-09-18T00:27:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:36.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary9 2024-09-18T00:27:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-18T00:27:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-18T00:27:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:27:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:27:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-18T00:27:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:27:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:27:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-18T00:27:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:27:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:27:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:27:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 10dacb73-f415-417d-8ed1-6bdb0fc5fcf2 2024-09-18T00:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":2051003.73,"bytes_per_snapshot":61530112.0,"last_snapshot_bytes":61530112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619197,"remote_snapshot_timestamp":1726619197,"replay_state":"idle"} 2024-09-18T00:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:17 2024-09-18T00:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:27:32' 2024-09-18T00:27:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:27:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:27:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:27:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":2051003.73,"bytes_per_snapshot":61530112.0,"last_snapshot_bytes":61530112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619197,"remote_snapshot_timestamp":1726619197,"replay_state":"idle"} 2024-09-18T00:27:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:27:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:27:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:27:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:27:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:27:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:27:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:27:17' 2024-09-18T00:27:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-18 00:27:17' 2024-09-18T00:27:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:27:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:27:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:27:17' 2024-09-18T00:27:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:27:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:27:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:27:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1010dacb73-f415-417d-8ed1-6bdb0fc5fcf2up+replayingreplaying, {"bytes_per_second":2051003.73,"bytes_per_snapshot":61530112.0,"last_snapshot_bytes":61530112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619197,"remote_snapshot_timestamp":1726619197,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:27:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:27:32' 2024-09-18T00:27:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:27:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:27:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:27:17' 2024-09-18T00:27:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:27:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:27:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:27:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:27:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:27:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-18T00:27:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:27:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:27:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-18T00:27:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-18T00:27:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:27:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-18T00:27:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:27:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:27:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:27:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary10 2024-09-18T00:27:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary10 2024-09-18T00:27:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..10} 2024-09-18T00:27:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS=() 2024-09-18T00:27:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS=() 2024-09-18T00:27:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS=() 2024-09-18T00:27:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS=() 2024-09-18T00:27:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ RET=0 2024-09-18T00:27:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-18T00:27:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary1 2024-09-18T00:27:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-18T00:27:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-18T00:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:27:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:27:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:27:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:27:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-18T00:27:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:27:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:27:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-18T00:27:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:27:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-18T00:27:39.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:27:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-18T00:27:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary1': 2024-09-18T00:27:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:27:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:27:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:27:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 3503ecbb23a0 2024-09-18T00:27:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.3503ecbb23a0 2024-09-18T00:27:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:27:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:27:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:27:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:27:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:27:39 2024 2024-09-18T00:27:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:27:39 2024 2024-09-18T00:27:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:27:39 2024 2024-09-18T00:27:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:27:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:27:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: ba910667-b6c2-40e5-972a-77379dacdcac 2024-09-18T00:27:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:27:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:27:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:27:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-18T00:27:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd0 2024-09-18T00:27:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:27:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-18T00:27:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:27:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:27:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:27:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: b149e144-1748-4f0a-857d-0a0a75b45c7f 2024-09-18T00:27:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:27:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:27:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:27:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:27:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:27:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:27:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary1 2024-09-18T00:27:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-18T00:27:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:27:40.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary1 2024-09-18T00:27:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:27:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-18T00:27:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary1 2024-09-18T00:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-18T00:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary2 2024-09-18T00:27:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-18T00:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary1 2024-09-18T00:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-18T00:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-18T00:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:27:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:27:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:27:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:27:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-18T00:27:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:27:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:27:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-18T00:27:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:27:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-18T00:27:40.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:27:40.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-18T00:27:40.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:27:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2646 2024-09-18T00:27:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-18T00:27:40.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary2': 2024-09-18T00:27:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:27:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:27:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:27:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 3510ffd4d42f 2024-09-18T00:27:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.3510ffd4d42f 2024-09-18T00:27:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:27:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:27:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:27:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:27:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:27:40 2024 2024-09-18T00:27:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:27:40 2024 2024-09-18T00:27:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:27:40 2024 2024-09-18T00:27:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:27:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:27:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: a8f733ff-3d00-42ab-a510-55f764b8b3ae 2024-09-18T00:27:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:27:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:27:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:27:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-18T00:27:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd1 2024-09-18T00:27:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:27:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-18T00:27:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:27:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:27:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:27:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 098db7a3-3e33-4789-8074-dfafd2cd22e9 2024-09-18T00:27:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:27:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:27:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:27:41.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:27:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:27:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:27:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary2 2024-09-18T00:27:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-18T00:27:41.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:27:41.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary2 2024-09-18T00:27:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:27:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-18T00:27:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:27:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:27:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:27:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:27:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:27:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary2 2024-09-18T00:27:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-18T00:27:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary3 2024-09-18T00:27:41.126 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-18T00:27:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary2 2024-09-18T00:27:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-18T00:27:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:41.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:27:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:27:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-18T00:27:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:27:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:27:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:27:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:27:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-18T00:27:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:27:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:27:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-18T00:27:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:27:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-18T00:27:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:27:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-18T00:27:41.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:27:41.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2648 2024-09-18T00:27:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-18T00:27:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary3': 2024-09-18T00:27:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:27:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:27:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:27:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 351f3da5bc5e 2024-09-18T00:27:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.351f3da5bc5e 2024-09-18T00:27:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:27:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:27:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:27:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:27:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:27:41 2024 2024-09-18T00:27:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:27:41 2024 2024-09-18T00:27:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:27:41 2024 2024-09-18T00:27:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:27:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:27:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 6c655549-4e7a-426d-9a67-53d1811be39f 2024-09-18T00:27:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:27:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:27:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:27:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-18T00:27:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd2 2024-09-18T00:27:42.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:27:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-18T00:27:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:27:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:27:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:27:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 7113af80-4737-47ab-b649-54734545d9e1 2024-09-18T00:27:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:27:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:27:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:27:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:27:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:27:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:27:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary3 2024-09-18T00:27:42.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-18T00:27:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:27:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary3 2024-09-18T00:27:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:27:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-18T00:27:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:27:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:27:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:27:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:27:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary3 2024-09-18T00:27:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:27:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-18T00:27:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary4 2024-09-18T00:27:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-18T00:27:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary3 2024-09-18T00:27:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-18T00:27:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:27:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:27:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-18T00:27:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:27:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:27:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:27:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:27:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-18T00:27:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:27:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:27:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-18T00:27:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:27:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-18T00:27:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:27:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-18T00:27:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:27:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-18T00:27:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary4': 2024-09-18T00:27:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:27:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:27:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:27:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 352e18cb9f3 2024-09-18T00:27:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.352e18cb9f3 2024-09-18T00:27:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:27:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:27:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:27:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:27:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:27:42 2024 2024-09-18T00:27:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:27:42 2024 2024-09-18T00:27:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:27:42 2024 2024-09-18T00:27:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:27:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:27:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 1eba87df-00c0-43f2-9499-35f1b3ee6e5f 2024-09-18T00:27:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:27:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:27:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:27:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-18T00:27:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd3 2024-09-18T00:27:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:27:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-18T00:27:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:27:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:27:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:27:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 6ae76dbe-e2a6-486a-8354-a23a5758472f 2024-09-18T00:27:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:27:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:27:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:27:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:27:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:27:43.203 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:27:43.202+0000 7f706e974640 -1 reset not still connected to 0x55df2eedb6c0 2024-09-18T00:27:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:27:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary4 2024-09-18T00:27:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-18T00:27:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:27:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary4 2024-09-18T00:27:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:27:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-18T00:27:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:27:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:27:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary4 2024-09-18T00:27:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:27:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-18T00:27:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:27:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary5 2024-09-18T00:27:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:27:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-18T00:27:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary4 2024-09-18T00:27:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-18T00:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-18T00:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:43.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:27:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:27:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:27:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:27:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-18T00:27:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:27:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:27:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-18T00:27:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:27:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-18T00:27:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:27:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-18T00:27:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:27:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:27:43.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:27:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:27:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2653 2024-09-18T00:27:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2652 2024-09-18T00:27:43.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-18T00:27:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary5': 2024-09-18T00:27:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:27:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:27:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:27:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 353c469f7661 2024-09-18T00:27:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.353c469f7661 2024-09-18T00:27:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:27:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:27:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:27:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:27:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:27:43 2024 2024-09-18T00:27:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:27:43 2024 2024-09-18T00:27:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:27:43 2024 2024-09-18T00:27:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:27:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:27:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9595931d-b126-4a67-bfbd-d2ac477b0311 2024-09-18T00:27:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:27:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:27:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:27:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-18T00:27:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd4 2024-09-18T00:27:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:27:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-18T00:27:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:27:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:27:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:27:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: b320921c-1a2e-46ad-9fe4-a4ae942d426e 2024-09-18T00:27:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:27:44.211 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:27:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:27:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:27:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:27:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:27:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary5 2024-09-18T00:27:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-18T00:27:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:27:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary5 2024-09-18T00:27:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:27:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-18T00:27:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:27:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:27:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-18T00:27:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary6 2024-09-18T00:27:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary5 2024-09-18T00:27:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-18T00:27:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:27:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:27:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:27:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:27:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:27:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary5 2024-09-18T00:27:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-18T00:27:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-18T00:27:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:27:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:27:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:27:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:27:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:44.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-18T00:27:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:27:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:27:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-18T00:27:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:44.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:27:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-18T00:27:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:27:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-18T00:27:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2656 2024-09-18T00:27:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:27:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-18T00:27:44.887 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:27:44.886+0000 7f0114255640 -1 reset not still connected to 0x557d0a065e10 2024-09-18T00:27:44.887 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:27:44.886+0000 7f0114255640 -1 reset not still connected to 0x557d0b465110 2024-09-18T00:27:44.890 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:27:44.888+0000 7f0114255640 -1 reset not still connected to 0x557d0b465860 2024-09-18T00:27:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:27:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:27:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:27:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary6': 2024-09-18T00:27:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:27:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:27:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:27:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 354e7170b394 2024-09-18T00:27:44.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.354e7170b394 2024-09-18T00:27:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:27:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:27:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:27:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:27:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:27:44 2024 2024-09-18T00:27:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:27:44 2024 2024-09-18T00:27:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:27:44 2024 2024-09-18T00:27:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:27:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:27:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 6dd62243-5126-440f-919f-f75472281ac7 2024-09-18T00:27:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:27:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:27:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:27:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-18T00:27:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd5 2024-09-18T00:27:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:27:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-18T00:27:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:27:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:27:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:27:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 610be618-f5ec-42cb-8234-526995a6e8d3 2024-09-18T00:27:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:27:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:27:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:27:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:27:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:27:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:27:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary6 2024-09-18T00:27:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-18T00:27:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:27:45.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary6 2024-09-18T00:27:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:27:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-18T00:27:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:27:45.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:27:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-18T00:27:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary6 2024-09-18T00:27:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:27:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary7 2024-09-18T00:27:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-18T00:27:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:27:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:27:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:27:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary6 2024-09-18T00:27:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:27:45.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-18T00:27:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-18T00:27:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:27:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:27:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:27:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:27:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-18T00:27:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:27:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:27:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-18T00:27:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:27:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:45.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-18T00:27:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:27:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-18T00:27:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:27:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-18T00:27:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2659 2024-09-18T00:27:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary7': 2024-09-18T00:27:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:27:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:27:46.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:27:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 355f3de59240 2024-09-18T00:27:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.355f3de59240 2024-09-18T00:27:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:27:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:27:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:27:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:27:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:27:45 2024 2024-09-18T00:27:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:27:45 2024 2024-09-18T00:27:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:27:45 2024 2024-09-18T00:27:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:27:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:27:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 688ff7b2-7bdc-4e47-87f8-0ebc17160572 2024-09-18T00:27:46.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:27:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:27:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:27:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-18T00:27:46.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd6 2024-09-18T00:27:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:27:46.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-18T00:27:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:27:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:27:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:27:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: c5410856-861f-4129-acce-6ced31ea6208 2024-09-18T00:27:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:27:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:27:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:27:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:27:46.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:27:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:27:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary7 2024-09-18T00:27:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-18T00:27:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:27:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary7 2024-09-18T00:27:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:27:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-18T00:27:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:27:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:27:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary7 2024-09-18T00:27:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-18T00:27:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary8 2024-09-18T00:27:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-18T00:27:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:27:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:27:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:27:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:27:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary7 2024-09-18T00:27:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:27:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-18T00:27:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-18T00:27:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:46.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:27:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:27:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:27:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:27:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:46.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-18T00:27:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:27:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:27:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-18T00:27:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:27:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:46.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-18T00:27:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:27:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-18T00:27:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2655 2024-09-18T00:27:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:27:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:27:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:27:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:27:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:27:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:27:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:27:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-18T00:27:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2661 2024-09-18T00:27:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary8': 2024-09-18T00:27:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:27:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:27:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:27:47.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 356e76212438 2024-09-18T00:27:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.356e76212438 2024-09-18T00:27:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:27:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:27:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:27:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:27:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:27:46 2024 2024-09-18T00:27:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:27:46 2024 2024-09-18T00:27:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:27:46 2024 2024-09-18T00:27:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:27:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:27:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: dfd928bc-d98b-418a-8961-dfb82037e5b1 2024-09-18T00:27:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:27:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:27:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:27:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-18T00:27:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd7 2024-09-18T00:27:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:27:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-18T00:27:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:27:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:27:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:27:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 0ea08a7b-703f-4bb4-b281-ac515385f192 2024-09-18T00:27:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:27:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:27:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:27:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:27:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:27:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:27:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary8 2024-09-18T00:27:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-18T00:27:47.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:27:47.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary8 2024-09-18T00:27:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:27:47.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-18T00:27:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:27:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:27:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary8 2024-09-18T00:27:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-18T00:27:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary9 2024-09-18T00:27:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-18T00:27:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:27:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:27:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:27:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary8 2024-09-18T00:27:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:27:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-18T00:27:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:27:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-18T00:27:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:27:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:27:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:27:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:27:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-18T00:27:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:27:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:27:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-18T00:27:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:27:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-18T00:27:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:27:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-18T00:27:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:27:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:27:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:27:47.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2658 2024-09-18T00:27:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:27:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2665 2024-09-18T00:27:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-18T00:27:48.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary9': 2024-09-18T00:27:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:27:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:27:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:27:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 3582d207eea7 2024-09-18T00:27:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.3582d207eea7 2024-09-18T00:27:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:27:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:27:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:27:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:27:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:27:47 2024 2024-09-18T00:27:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:27:47 2024 2024-09-18T00:27:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:27:47 2024 2024-09-18T00:27:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:27:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:27:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15 2024-09-18T00:27:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:27:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:27:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:27:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-18T00:27:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd8 2024-09-18T00:27:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:27:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-18T00:27:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:27:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:27:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:27:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: b87739f7-ac45-4a03-829c-64afd1ac9734 2024-09-18T00:27:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:27:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:27:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:27:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:27:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:27:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:27:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary9 2024-09-18T00:27:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-18T00:27:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:27:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary9 2024-09-18T00:27:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:27:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-18T00:27:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:27:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:27:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:27:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:27:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-18T00:27:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:27:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary9 2024-09-18T00:27:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary10 2024-09-18T00:27:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:27:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-18T00:27:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:27:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:27:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary9 2024-09-18T00:27:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-18T00:27:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-18T00:27:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:27:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:27:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:27:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:27:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:27:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-18T00:27:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:27:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:27:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:27:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-18T00:27:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:27:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:27:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-18T00:27:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:27:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-18T00:27:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2664 2024-09-18T00:27:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:27:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:27:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:27:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:27:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-18T00:27:49.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2668 2024-09-18T00:27:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:49.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary10': 2024-09-18T00:27:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:27:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:27:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:27:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 3591623a195c 2024-09-18T00:27:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.3591623a195c 2024-09-18T00:27:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:27:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:27:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:27:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:27:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:27:48 2024 2024-09-18T00:27:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:27:48 2024 2024-09-18T00:27:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:27:48 2024 2024-09-18T00:27:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:27:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:27:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 3f8781e9-128d-405e-83d4-3ab3eade1052 2024-09-18T00:27:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:27:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:27:49.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:27:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-18T00:27:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd9 2024-09-18T00:27:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:27:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-18T00:27:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:27:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:27:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:27:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 97c1d96e-1d05-40c2-9712-d45574f6d4e1 2024-09-18T00:27:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:27:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:27:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:27:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:27:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:27:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:27:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:27:49.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary10 2024-09-18T00:27:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-18T00:27:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:27:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary10 2024-09-18T00:27:49.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-18T00:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary10 2024-09-18T00:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 130255 2024-09-18T00:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:27:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:27:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary10 2024-09-18T00:27:49.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-18T00:27:49.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:27:49.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-18T00:27:49.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2663 2024-09-18T00:27:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2667 2024-09-18T00:27:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:49.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2671 2024-09-18T00:27:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:27:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:27:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:27:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:27:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:27:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:27:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2670 2024-09-18T00:27:50.908 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:27:50.907+0000 7f9bfab49640 -1 reset not still connected to 0x55cd4a4ac820 2024-09-18T00:27:50.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:27:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:27:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:27:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2672 2024-09-18T00:27:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:27:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:27:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:27:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2673 2024-09-18T00:27:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:51.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2674 2024-09-18T00:27:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:27:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:51.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:27:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:27:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:27:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:27:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:27:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:27:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:27:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:27:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:27:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:27:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:27:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:27:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:27:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:27:52.912 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:27:52.910+0000 7f6acfc5f640 -1 reset not still connected to 0x555c34819040 2024-09-18T00:27:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2676 2024-09-18T00:27:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:27:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:27:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:27:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2675 2024-09-18T00:27:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:27:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:27:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:27:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2678 2024-09-18T00:27:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2677 2024-09-18T00:27:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2680 2024-09-18T00:27:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:27:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:27:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:27:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:27:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:27:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:27:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2682 2024-09-18T00:27:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2679 2024-09-18T00:27:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2681 2024-09-18T00:27:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2683 2024-09-18T00:27:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2684 2024-09-18T00:27:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:27:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:27:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:27:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:27:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:27:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:27:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:27:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:27:57.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:27:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:27:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:27:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:27:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:27:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:27:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:27:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2686 2024-09-18T00:27:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:27:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:27:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:27:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2687 2024-09-18T00:27:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:27:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:27:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:27:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:27:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:27:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:27:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2688 2024-09-18T00:27:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2685 2024-09-18T00:27:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2689 2024-09-18T00:27:59.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:59.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:27:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:27:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:27:59.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:27:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:27:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:27:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:27:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:27:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:27:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:28:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2692 2024-09-18T00:28:00.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2690 2024-09-18T00:28:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:28:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:28:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:28:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2693 2024-09-18T00:28:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2694 2024-09-18T00:28:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2691 2024-09-18T00:28:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:28:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:28:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:28:02.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:28:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:28:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:28:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:28:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:28:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:28:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:28:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:28:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:28:02.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2695 2024-09-18T00:28:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2697 2024-09-18T00:28:03.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2696 2024-09-18T00:28:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2699 2024-09-18T00:28:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:28:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:28:04.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:28:04.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:28:04.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:28:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:28:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:28:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:28:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:28:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:28:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:28:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:28:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:28:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:28:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:28:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2700 2024-09-18T00:28:05.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2701 2024-09-18T00:28:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2698 2024-09-18T00:28:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:05.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2702 2024-09-18T00:28:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:28:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:06.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:28:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:28:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2704 2024-09-18T00:28:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2703 2024-09-18T00:28:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:28:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:28:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:28:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:28:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:28:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:28:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:28:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:28:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:28:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2705 2024-09-18T00:28:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:28:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:28:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:28:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:28:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:28:08.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:28:08.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:28:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:28:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:28:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:28:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2708 2024-09-18T00:28:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2706 2024-09-18T00:28:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2707 2024-09-18T00:28:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:09.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:09.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2711 2024-09-18T00:28:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:28:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:28:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:28:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:28:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:28:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:28:10.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2710 2024-09-18T00:28:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2712 2024-09-18T00:28:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2709 2024-09-18T00:28:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:28:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:28:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:28:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2714 2024-09-18T00:28:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2713 2024-09-18T00:28:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:28:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:28:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:28:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2715 2024-09-18T00:28:12.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:28:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:28:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:28:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:28:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:28:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:28:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:28:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:28:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:28:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2716 2024-09-18T00:28:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:28:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:28:13.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:28:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2717 2024-09-18T00:28:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:28:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:28:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2718 2024-09-18T00:28:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:13.912 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:28:13.909+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4340 2024-09-18T00:28:13.913 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:28:13.911+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb52b0 2024-09-18T00:28:13.914 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:28:13.911+0000 7f706e974640 -1 reset not still connected to 0x55df22fb9a00 2024-09-18T00:28:13.916 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:28:13.913+0000 7f706e974640 -1 reset not still connected to 0x55df2bfddad0 2024-09-18T00:28:13.916 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:28:13.913+0000 7f706e974640 -1 reset not still connected to 0x55df4fdad520 2024-09-18T00:28:13.918 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:28:13.915+0000 7f706e974640 -1 reset not still connected to 0x55df55c6d380 2024-09-18T00:28:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2719 2024-09-18T00:28:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:28:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:28:14.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:28:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:28:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:28:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:28:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2720 2024-09-18T00:28:15.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2721 2024-09-18T00:28:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2722 2024-09-18T00:28:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:28:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:28:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:28:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:16.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2723 2024-09-18T00:28:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2724 2024-09-18T00:28:16.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:16.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:16.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:28:16.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:16.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:16.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:28:16.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:28:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:28:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:16.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:28:16.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:28:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:28:16.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:28:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:28:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2725 2024-09-18T00:28:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:28:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:28:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:28:17.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:17.924 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:28:17.922+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4dd0 2024-09-18T00:28:17.924 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:28:17.922+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ea270 2024-09-18T00:28:17.924 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:28:17.922+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:28:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2728 2024-09-18T00:28:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2727 2024-09-18T00:28:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:28:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:28:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:28:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:28:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:28:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:28:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:28:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:28:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:28:19.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2729 2024-09-18T00:28:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2726 2024-09-18T00:28:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:28:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:28:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:19.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:28:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:28:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:28:20.447 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:28:20.445+0000 7f706e974640 -1 reset not still connected to 0x55df26543520 2024-09-18T00:28:20.447 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:28:20.445+0000 7f706e974640 -1 reset not still connected to 0x55df2eedb6c0 2024-09-18T00:28:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2730 2024-09-18T00:28:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:28:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:28:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:28:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2731 2024-09-18T00:28:20.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:21.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2732 2024-09-18T00:28:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2733 2024-09-18T00:28:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:28:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:28:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:28:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:28:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:28:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:28:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:28:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:28:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:28:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2734 2024-09-18T00:28:22.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2735 2024-09-18T00:28:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:28:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:28:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:28:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:22.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:23.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2737 2024-09-18T00:28:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2736 2024-09-18T00:28:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:28:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:23.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:28:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:28:23.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:28:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:28:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:28:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2738 2024-09-18T00:28:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:24.227 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:28:24.225+0000 7fc275b74640 -1 reset not still connected to 0x559cc572e750 2024-09-18T00:28:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2739 2024-09-18T00:28:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:28:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:28:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:28:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:28:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:28:24.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:28:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2741 2024-09-18T00:28:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2740 2024-09-18T00:28:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:28:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:28:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:28:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:28:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:28:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:28:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2743 2024-09-18T00:28:25.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2742 2024-09-18T00:28:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:28:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:28:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:28:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2745 2024-09-18T00:28:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:28:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:28:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:28:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:28:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:28:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:28:27.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:28:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:28:27.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:28:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2744 2024-09-18T00:28:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:28.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2746 2024-09-18T00:28:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2747 2024-09-18T00:28:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:28:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:28:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:28:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:28:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:28:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:28:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:28:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:28:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:28:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:29.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2748 2024-09-18T00:28:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:29.909 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:28:29.907+0000 7f9bfab49640 -1 reset not still connected to 0x55cd48520000 2024-09-18T00:28:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:28:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:28:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:28:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:28:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:28:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:28:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2752 2024-09-18T00:28:30.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2749 2024-09-18T00:28:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2750 2024-09-18T00:28:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:28:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:28:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:28:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2751 2024-09-18T00:28:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2754 2024-09-18T00:28:31.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2753 2024-09-18T00:28:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:28:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:31.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:28:31.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:28:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:28:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:28:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:28:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:28:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:28:32.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:28:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2755 2024-09-18T00:28:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:33.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2757 2024-09-18T00:28:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2756 2024-09-18T00:28:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:28:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:28:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:28:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:28:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:28:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:28:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:28:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:28:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:28:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2758 2024-09-18T00:28:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2761 2024-09-18T00:28:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:28:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:28:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:28:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:28:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:28:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:28:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:28:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:28:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:28:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2759 2024-09-18T00:28:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2762 2024-09-18T00:28:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:28:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:28:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:28:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2760 2024-09-18T00:28:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2764 2024-09-18T00:28:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2763 2024-09-18T00:28:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:28:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:28:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:28:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:28:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:28:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:28:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:28:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:28:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:28:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:28:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:28:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:28:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2765 2024-09-18T00:28:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:28:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:28:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:28:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2766 2024-09-18T00:28:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2768 2024-09-18T00:28:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2767 2024-09-18T00:28:38.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:38.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:28:38.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:38.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:38.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:28:38.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:28:38.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:38.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:28:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:28:39.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:28:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:28:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:28:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:28:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:28:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:28:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:28:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2772 2024-09-18T00:28:40.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2769 2024-09-18T00:28:40.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2771 2024-09-18T00:28:40.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:40.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2774 2024-09-18T00:28:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:28:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:28:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:28:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2775 2024-09-18T00:28:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2773 2024-09-18T00:28:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:28:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:28:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:28:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:41.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:28:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:28:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:28:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:28:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:28:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:28:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2776 2024-09-18T00:28:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:28:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:28:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:28:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:28:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:28:43.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:28:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:28:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:28:43.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:28:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:28:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:28:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:28:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2779 2024-09-18T00:28:43.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2777 2024-09-18T00:28:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:44.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2778 2024-09-18T00:28:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2781 2024-09-18T00:28:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:28:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:28:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:28:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:28:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:28:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:28:45.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2783 2024-09-18T00:28:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2782 2024-09-18T00:28:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2780 2024-09-18T00:28:45.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:28:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:28:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:28:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2785 2024-09-18T00:28:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2784 2024-09-18T00:28:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:28:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:28:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:28:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:28:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:28:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:28:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2786 2024-09-18T00:28:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:28:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:28:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:28:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:28:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:28:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:28:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2787 2024-09-18T00:28:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2788 2024-09-18T00:28:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:28:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:28:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:28:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:28:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:28:48.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:28:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:28:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:28:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:28:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2789 2024-09-18T00:28:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:49.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2790 2024-09-18T00:28:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:28:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:28:49.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:28:49.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:28:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:28:49.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:28:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2793 2024-09-18T00:28:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:28:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:28:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:28:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2791 2024-09-18T00:28:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:51.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2792 2024-09-18T00:28:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2795 2024-09-18T00:28:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2794 2024-09-18T00:28:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:28:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:28:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:28:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:51.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:51.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:28:51.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:51.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:51.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:28:51.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:28:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:28:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:28:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:28:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:51.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2796 2024-09-18T00:28:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:28:52.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:28:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:28:52.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2798 2024-09-18T00:28:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:28:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:28:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:28:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2797 2024-09-18T00:28:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:28:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:28:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:28:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2799 2024-09-18T00:28:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2800 2024-09-18T00:28:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:28:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:28:54.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:28:54.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:28:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:28:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:28:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:28:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:28:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:28:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2802 2024-09-18T00:28:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2801 2024-09-18T00:28:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:28:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:28:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:28:55.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2804 2024-09-18T00:28:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2805 2024-09-18T00:28:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2803 2024-09-18T00:28:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:28:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:28:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:28:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:28:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:28:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:28:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:56.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:28:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:28:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:28:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2806 2024-09-18T00:28:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:28:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:28:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:28:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2807 2024-09-18T00:28:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2808 2024-09-18T00:28:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:28:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:28:58.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:28:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:28:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:28:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:28:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2810 2024-09-18T00:28:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:28:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:28:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:28:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:59.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2809 2024-09-18T00:28:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:28:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:28:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:28:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:28:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:28:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:28:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:28:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:28:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:28:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:29:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2812 2024-09-18T00:29:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2811 2024-09-18T00:29:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:29:00.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:29:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:29:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2813 2024-09-18T00:29:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2814 2024-09-18T00:29:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2815 2024-09-18T00:29:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:29:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:29:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:29:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:29:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:29:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:29:01.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:01.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:29:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:29:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:29:01.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:29:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:29:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:29:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2816 2024-09-18T00:29:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:03.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2817 2024-09-18T00:29:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2818 2024-09-18T00:29:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:29:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:29:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:29:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:29:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:29:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:29:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:03.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2819 2024-09-18T00:29:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2820 2024-09-18T00:29:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:04.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:04.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:29:04.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:29:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:29:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:29:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:29:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:29:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:29:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:29:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:29:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2821 2024-09-18T00:29:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:29:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:29:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:29:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2823 2024-09-18T00:29:06.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:06.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2822 2024-09-18T00:29:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2824 2024-09-18T00:29:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:29:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:29:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:29:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:06.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:29:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:29:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:29:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:29:07.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:29:07.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:29:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:29:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:29:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:29:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2826 2024-09-18T00:29:07.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2825 2024-09-18T00:29:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2827 2024-09-18T00:29:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:29:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:29:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:29:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2830 2024-09-18T00:29:08.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:29:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:29:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:29:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2829 2024-09-18T00:29:09.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2828 2024-09-18T00:29:09.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:09.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:29:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:29:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:29:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:29:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:29:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:29:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2831 2024-09-18T00:29:10.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2832 2024-09-18T00:29:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:29:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:29:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:29:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:29:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:29:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:29:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2833 2024-09-18T00:29:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2834 2024-09-18T00:29:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:29:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:29:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:29:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2835 2024-09-18T00:29:11.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:29:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:29:11.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:29:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:29:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:29:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:29:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:29:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:29:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:29:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2836 2024-09-18T00:29:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2837 2024-09-18T00:29:13.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2838 2024-09-18T00:29:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:29:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:29:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:29:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:29:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:29:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:29:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:29:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:29:14.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:29:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2839 2024-09-18T00:29:14.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:14.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2840 2024-09-18T00:29:14.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:29:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:29:14.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:29:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:29:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:29:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:29:15.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2841 2024-09-18T00:29:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2843 2024-09-18T00:29:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:29:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:29:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:29:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2845 2024-09-18T00:29:16.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2842 2024-09-18T00:29:16.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2844 2024-09-18T00:29:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:29:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:29:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:29:17.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:29:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:29:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:29:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:29:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:29:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:29:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2846 2024-09-18T00:29:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:29:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:29:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:29:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:29:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:29:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:29:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2849 2024-09-18T00:29:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:29:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:29:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:29:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2848 2024-09-18T00:29:18.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:19.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2847 2024-09-18T00:29:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2850 2024-09-18T00:29:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:29:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:29:19.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:29:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:29:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:29:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:29:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:29:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:29:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:29:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:20.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2851 2024-09-18T00:29:20.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2852 2024-09-18T00:29:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:29:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:29:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:29:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2853 2024-09-18T00:29:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2855 2024-09-18T00:29:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2854 2024-09-18T00:29:21.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:29:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:29:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:29:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:29:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:29:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:29:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2856 2024-09-18T00:29:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:29:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:29:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:29:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:29:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:29:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:29:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2857 2024-09-18T00:29:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:29:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:29:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:29:23.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:29:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:29:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:29:23.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:23.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2859 2024-09-18T00:29:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:29:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:29:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:29:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2858 2024-09-18T00:29:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2860 2024-09-18T00:29:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:29:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:29:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:29:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:29:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:29:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:29:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2862 2024-09-18T00:29:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:29:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:29:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:29:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2863 2024-09-18T00:29:25.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2861 2024-09-18T00:29:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2864 2024-09-18T00:29:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:29:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:29:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:29:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:29:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:26.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:26.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:29:26.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:29:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2866 2024-09-18T00:29:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:29:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:29:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:29:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:29:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:29:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:29:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2865 2024-09-18T00:29:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:27.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2868 2024-09-18T00:29:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2867 2024-09-18T00:29:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:29:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:28.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:29:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:29:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:29:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:28.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:29:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:29:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2869 2024-09-18T00:29:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:29:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:29:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:29:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2870 2024-09-18T00:29:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:29:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:29:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:29:30.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2872 2024-09-18T00:29:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2873 2024-09-18T00:29:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2871 2024-09-18T00:29:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:29:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:29:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:29:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:29:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:29:30.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:29:30.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2874 2024-09-18T00:29:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:29:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:29:31.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:29:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:29:31.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:29:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:29:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2876 2024-09-18T00:29:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:29:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:29:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:29:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:29:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:29:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:29:32.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2875 2024-09-18T00:29:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2879 2024-09-18T00:29:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2878 2024-09-18T00:29:33.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2877 2024-09-18T00:29:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:29:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:29:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:29:33.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:29:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:29:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:29:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:29:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:29:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:29:33.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2880 2024-09-18T00:29:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:29:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:29:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:29:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:29:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2882 2024-09-18T00:29:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2883 2024-09-18T00:29:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2881 2024-09-18T00:29:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:29:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:29:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:29:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2884 2024-09-18T00:29:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2885 2024-09-18T00:29:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:29:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:29:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:29:36.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:36.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:29:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:29:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:29:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:29:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:29:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:29:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:29:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2886 2024-09-18T00:29:37.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:29:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:29:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:29:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:29:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:38.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:29:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:29:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:29:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:29:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:29:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2889 2024-09-18T00:29:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2887 2024-09-18T00:29:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:38.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:39.010 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:29:39.008+0000 7f9bfab49640 -1 reset not still connected to 0x55cd434c0c30 2024-09-18T00:29:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2890 2024-09-18T00:29:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2888 2024-09-18T00:29:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:29:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:29:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:29:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:29:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:29:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:29:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2892 2024-09-18T00:29:40.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2893 2024-09-18T00:29:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:29:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:29:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:29:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2891 2024-09-18T00:29:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2894 2024-09-18T00:29:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2895 2024-09-18T00:29:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:41.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:29:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:29:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:29:41.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:41.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:29:41.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:41.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:41.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:29:41.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:29:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:29:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:29:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:29:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:29:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2896 2024-09-18T00:29:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:43.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2897 2024-09-18T00:29:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2898 2024-09-18T00:29:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:29:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:29:43.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:29:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:29:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:29:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:29:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:29:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:29:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:29:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:44.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2899 2024-09-18T00:29:44.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2900 2024-09-18T00:29:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:45.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:29:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:29:45.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:29:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:29:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:29:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:29:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2901 2024-09-18T00:29:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2903 2024-09-18T00:29:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:29:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:29:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:29:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2904 2024-09-18T00:29:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2905 2024-09-18T00:29:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2902 2024-09-18T00:29:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:29:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:29:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:29:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:29:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:29:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:29:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:29:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:29:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:29:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:29:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:29:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:29:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2906 2024-09-18T00:29:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:29:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:29:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:29:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:29:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:29:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:29:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2908 2024-09-18T00:29:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2909 2024-09-18T00:29:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2907 2024-09-18T00:29:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:49.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2910 2024-09-18T00:29:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:29:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:50.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:29:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:29:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:29:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:29:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:29:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:29:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:29:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:29:50.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2911 2024-09-18T00:29:50.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2912 2024-09-18T00:29:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:50.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:50.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2915 2024-09-18T00:29:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:29:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:29:51.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2914 2024-09-18T00:29:51.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2913 2024-09-18T00:29:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:51.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:29:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:29:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:29:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:29:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:29:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:29:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:29:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:29:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:29:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2916 2024-09-18T00:29:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:29:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:29:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:29:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2919 2024-09-18T00:29:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2918 2024-09-18T00:29:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2917 2024-09-18T00:29:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:29:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:29:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:29:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:29:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:29:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:29:53.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:29:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:29:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:29:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2920 2024-09-18T00:29:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:29:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:29:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:29:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:29:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:29:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:29:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2922 2024-09-18T00:29:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2921 2024-09-18T00:29:55.867 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2923 2024-09-18T00:29:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:29:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:29:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:29:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2925 2024-09-18T00:29:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2924 2024-09-18T00:29:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:29:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:29:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:29:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:29:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:29:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:29:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:29:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:29:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:29:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2926 2024-09-18T00:29:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:29:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:29:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:29:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:58.873 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2928 2024-09-18T00:29:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2929 2024-09-18T00:29:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2927 2024-09-18T00:29:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:29:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:29:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:29:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:29:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:29:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:29:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:29:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:29:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:29:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:29:59.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2930 2024-09-18T00:29:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:29:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:29:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:29:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:29:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:29:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:29:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:29:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:29:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:29:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2931 2024-09-18T00:30:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:30:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:30:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:30:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2932 2024-09-18T00:30:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2933 2024-09-18T00:30:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:01.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2935 2024-09-18T00:30:01.899 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2934 2024-09-18T00:30:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:30:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:01.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:30:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:30:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:30:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:30:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:30:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:30:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:30:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:30:01.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2936 2024-09-18T00:30:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:30:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:30:02.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:30:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2937 2024-09-18T00:30:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2938 2024-09-18T00:30:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2939 2024-09-18T00:30:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:30:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:30:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:30:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:30:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:30:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:30:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:30:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:30:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:30:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2940 2024-09-18T00:30:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:30:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:30:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:30:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:30:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:30:04.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:30:05.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2941 2024-09-18T00:30:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2942 2024-09-18T00:30:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2943 2024-09-18T00:30:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:30:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:30:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:30:05.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:05.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:05.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2944 2024-09-18T00:30:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2945 2024-09-18T00:30:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:30:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:30:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:30:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:30:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:30:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:30:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:30:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:30:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:30:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:30:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:30:07.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:30:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2946 2024-09-18T00:30:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2949 2024-09-18T00:30:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2947 2024-09-18T00:30:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2948 2024-09-18T00:30:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:30:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 130387 2024-09-18T00:30:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:30:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:30:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:30:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:30:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:30:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:30:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:09.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2950 2024-09-18T00:30:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:30:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:30:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:30:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:30:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 130519 2024-09-18T00:30:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2951 2024-09-18T00:30:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2952 2024-09-18T00:30:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:30:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:30:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:30:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:10.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2953 2024-09-18T00:30:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:30:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 130650 2024-09-18T00:30:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:30:11.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 130803 2024-09-18T00:30:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:30:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:30:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:30:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:30:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:30:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:30:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:30:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:30:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:30:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2954 2024-09-18T00:30:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2955 2024-09-18T00:30:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:30:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 130955 2024-09-18T00:30:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:30:13.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 131087 2024-09-18T00:30:13.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:14.857 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2956 2024-09-18T00:30:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:30:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:30:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 131260 2024-09-18T00:30:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:30:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 131413 2024-09-18T00:30:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:30:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 131567 2024-09-18T00:30:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:30:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:30:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 130256 2024-09-18T00:32:36.023 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:32:36.020+0000 7f9bfab49640 -1 reset not still connected to 0x55cd434c0c30 2024-09-18T00:32:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:32:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:32:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:32:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 130388 2024-09-18T00:32:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:32:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:32:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:32:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 130520 2024-09-18T00:32:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:32:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:32:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:32:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 130651 2024-09-18T00:32:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:32:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:32:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:32:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 130804 2024-09-18T00:32:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:32:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:32:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:32:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 130956 2024-09-18T00:32:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:32:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:32:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:32:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 131088 2024-09-18T00:32:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:32:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:32:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:32:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 131261 2024-09-18T00:32:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:32:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:32:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:32:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 131414 2024-09-18T00:32:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:32:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:32:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:32:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 131568 2024-09-18T00:32:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:32:49.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:32:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:32:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:32:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:32:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:32:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 1 2024-09-18T00:32:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd0 2024-09-18T00:32:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:32:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary1 2024-09-18T00:32:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary1 2024-09-18T00:32:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:32:49.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary1 2024-09-18T00:32:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:32:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 2 2024-09-18T00:32:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd1 2024-09-18T00:32:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary2 2024-09-18T00:32:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:32:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary2 2024-09-18T00:32:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:32:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary2 2024-09-18T00:32:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 3 2024-09-18T00:32:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:32:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd2 2024-09-18T00:32:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary3 2024-09-18T00:32:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary3 2024-09-18T00:32:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:32:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary3 2024-09-18T00:32:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:32:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 4 2024-09-18T00:32:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:32:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd3 2024-09-18T00:32:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary4 2024-09-18T00:32:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary4 2024-09-18T00:32:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:32:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:32:49.656 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary4 2024-09-18T00:32:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:32:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 5 2024-09-18T00:32:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd4 2024-09-18T00:32:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:32:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary5 2024-09-18T00:32:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary5 2024-09-18T00:32:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:32:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:32:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary5 2024-09-18T00:32:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 6 2024-09-18T00:32:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd5 2024-09-18T00:32:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary6 2024-09-18T00:32:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:32:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary6 2024-09-18T00:32:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:32:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary6 2024-09-18T00:32:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:32:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 7 2024-09-18T00:32:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:32:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd6 2024-09-18T00:32:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary7 2024-09-18T00:32:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:32:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary7 2024-09-18T00:32:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:32:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary7 2024-09-18T00:32:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:32:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:32:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:32:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:32:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 138501 2024-09-18T00:32:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 9 2024-09-18T00:32:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd8 2024-09-18T00:32:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary9 2024-09-18T00:32:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 8 2024-09-18T00:32:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary9 2024-09-18T00:32:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:32:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary9 2024-09-18T00:32:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd7 2024-09-18T00:32:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary8 2024-09-18T00:32:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 10 2024-09-18T00:32:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary8 2024-09-18T00:32:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:32:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary8 2024-09-18T00:32:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd9 2024-09-18T00:32:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary10 2024-09-18T00:32:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary10 2024-09-18T00:32:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:32:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary10 2024-09-18T00:32:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-18T00:32:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:32:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-18T00:32:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:32:50.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-18T00:32:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:32:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-18T00:32:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:32:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:32:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-18T00:32:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-18T00:32:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:32:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-18T00:32:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:32:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-18T00:32:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:32:50.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-18T00:32:50.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:32:50.144 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:32:50.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-18T00:33:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=555a93fbc64edf9288910e7a86060018 2024-09-18T00:33:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-18T00:33:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-18T00:33:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:33:20.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-18T00:33:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=1e573f91c70a1dfdddd1118a7f8e7547 2024-09-18T00:33:20.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-18T00:33:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=395b4d5d06656c3b34a21075aaa327a8 2024-09-18T00:33:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-18T00:33:20.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-18T00:33:20.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:20.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:33:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-18T00:33:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-18T00:33:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:33:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-18T00:33:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=3dbd85a9bd05a06145e3f4f2fad165b6 2024-09-18T00:33:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-18T00:33:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-18T00:33:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:33:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-18T00:33:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=bfd99bccb242d7e858c15a109dd3d5e2 2024-09-18T00:33:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-18T00:33:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=16e6707b68e924fc529bd4cfe118229c 2024-09-18T00:33:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-18T00:33:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-18T00:33:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:33:21.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-18T00:33:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=c1c758bc9322e346c3489ebb0a36c5ec 2024-09-18T00:33:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-18T00:33:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-18T00:33:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:33:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-18T00:33:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=bd980d751b9a15c8decdf4fc50bd64f6 2024-09-18T00:33:21.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-18T00:33:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-18T00:33:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:33:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-18T00:33:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-18T00:33:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:33:21.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-18T00:33:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=1a7a3ebad130cc0996c97ffb3e6bdbc0 2024-09-18T00:33:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-18T00:33:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=f93a51c035317657007be9fc31fa160d 2024-09-18T00:33:21.530 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-18T00:33:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-18T00:33:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:33:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-18T00:33:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-18T00:33:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:33:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-18T00:33:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:33:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-18T00:33:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:33:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:33:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-18T00:33:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:33:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:33:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:33:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:33:23.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:33:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:33:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:33:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:33:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-18T00:33:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:33:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:23.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-18T00:33:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:33:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:23.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-18T00:33:23.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:23.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:23.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:33:23.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:33:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:33:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-18T00:33:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:33:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-18T00:33:23.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:33:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-18T00:33:23.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:33:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:23.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-18T00:33:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-18T00:33:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:33:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:33:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:23.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-18T00:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:23.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:33:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: dfd928bc-d98b-418a-8961-dfb82037e5b1 2024-09-18T00:33:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22 2024-09-18T00:33:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:33:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:33:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22' 2024-09-18T00:33:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:23.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:33:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-18 00:33:22' 2024-09-18T00:33:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:33:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:22' 2024-09-18T00:33:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1591296.0,"last_snapshot_bytes":3022848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619404,"remote_snapshot_timestamp":1726619404,"replay_state":"idle"}2024-09-18 00:33:17image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619413,"remote_snapshot_timestamp":1726619413,"replay_state":"idle"}2024-09-18 00:33:17image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3387392.0,"last_snapshot_bytes":4780032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619405,"remote_snapshot_timestamp":1726619405,"replay_state":"idle"}2024-09-18 00:33:17image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3704832.0,"last_snapshot_bytes":675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619407,"remote_snapshot_timestamp":1726619407,"replay_state":"idle"}2024-09-18 00:33:17image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3368960.0,"last_snapshot_bytes":126976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619407,"remote_snapshot_timestamp":1726619407,"replay_state":"idle"}2024-09-18 00:33:17image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3639296.0,"last_snapshot_bytes":3887104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619409,"remote_snapshot_timestamp":1726619409,"replay_state":"idle"}2024-09-18 00:33:17image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3608576.0,"last_snapshot_bytes":3129344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619409,"remote_snapshot_timestamp":1726619409,"replay_state":"idle"}2024-09-18 00:33:17image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619410,"remote_snapshot_timestamp":1726619410,"replay_state":"idle"}2024-09-18 00:33:17image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2631680.0,"last_snapshot_bytes":5120000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619412,"remote_snapshot_timestamp":1726619412,"replay_state":"idle"}2024-09-18 00:33:17' 2024-09-18T00:33:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:33:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:33:23.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:23.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:23.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-18T00:33:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:33:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:23.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:23.584 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:33:23.582+0000 7fc275b74640 -1 reset not still connected to 0x559cc0d57860 2024-09-18T00:33:23.584 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:33:23.582+0000 7fc275b74640 -1 reset not still connected to 0x559cc45ad450 2024-09-18T00:33:23.584 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:33:23.582+0000 7fc275b74640 -1 reset not still connected to 0x559cc47c0270 2024-09-18T00:33:23.584 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:33:23.582+0000 7fc275b74640 -1 reset not still connected to 0x559cc572e750 2024-09-18T00:33:24.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-18T00:33:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:33:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:33:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-18T00:33:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:33:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:33:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-18T00:33:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:33:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:33:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-18T00:33:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:33:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-18T00:33:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:33:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:33:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-18T00:33:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-18T00:33:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:33:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:33:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-18T00:33:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:33:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-18T00:33:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:33:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-18T00:33:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:33:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:33:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-18T00:33:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:33:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:33:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9595931d-b126-4a67-bfbd-d2ac477b0311 2024-09-18T00:33:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22 2024-09-18T00:33:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:33:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:33:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3639296.0,"last_snapshot_bytes":3887104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619409,"remote_snapshot_timestamp":1726619409,"replay_state":"idle"} 2024-09-18T00:33:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:17' 2024-09-18T00:33:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:33:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:33:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 3f8781e9-128d-405e-83d4-3ab3eade1052 2024-09-18T00:33:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22 2024-09-18T00:33:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:33:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:33:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619413,"remote_snapshot_timestamp":1726619413,"replay_state":"idle"} 2024-09-18T00:33:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:17' 2024-09-18T00:33:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:24.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:24.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:33:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ba910667-b6c2-40e5-972a-77379dacdcac 2024-09-18T00:33:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22 2024-09-18T00:33:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:33:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:33:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1591296.0,"last_snapshot_bytes":3022848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619404,"remote_snapshot_timestamp":1726619404,"replay_state":"idle"} 2024-09-18T00:33:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-18 00:33:22' 2024-09-18T00:33:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:17' 2024-09-18T00:33:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:33:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:22' 2024-09-18T00:33:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:33:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:33:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:24.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:24.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-18 00:33:22' 2024-09-18T00:33:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:33:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:22' 2024-09-18T00:33:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:33:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 6c655549-4e7a-426d-9a67-53d1811be39f 2024-09-18T00:33:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22 2024-09-18T00:33:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:33:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:33:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3704832.0,"last_snapshot_bytes":675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619407,"remote_snapshot_timestamp":1726619407,"replay_state":"idle"} 2024-09-18T00:33:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:17' 2024-09-18T00:33:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:24.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-18 00:33:22' 2024-09-18T00:33:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:33:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:22' 2024-09-18T00:33:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:24.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:33:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 6dd62243-5126-440f-919f-f75472281ac7 2024-09-18T00:33:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22 2024-09-18T00:33:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:33:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:33:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:24' 2024-09-18T00:33:24.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:33:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1eba87df-00c0-43f2-9499-35f1b3ee6e5f 2024-09-18T00:33:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22 2024-09-18T00:33:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:33:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopping_replay 2024-09-18T00:33:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:33:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:23' 2024-09-18T00:33:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:33:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:24.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:24.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:24.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:24.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:33:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:33:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:33:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a8f733ff-3d00-42ab-a510-55f764b8b3ae 2024-09-18T00:33:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22 2024-09-18T00:33:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:33:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopping_replay 2024-09-18T00:33:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:33:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:24' 2024-09-18T00:33:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:24.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-18 00:33:22' 2024-09-18T00:33:24.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:33:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:22' 2024-09-18T00:33:24.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-18 00:33:22' 2024-09-18T00:33:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:33:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:22' 2024-09-18T00:33:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:33:24.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:33:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 688ff7b2-7bdc-4e47-87f8-0ebc17160572 2024-09-18T00:33:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22 2024-09-18T00:33:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:33:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopping_replay 2024-09-18T00:33:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:33:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:24' 2024-09-18T00:33:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:24.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-18 00:33:22' 2024-09-18T00:33:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:33:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:22' 2024-09-18T00:33:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:33:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:24.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-18 00:33:22' 2024-09-18T00:33:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:33:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:22' 2024-09-18T00:33:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-18 00:33:22' 2024-09-18T00:33:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:33:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:22' 2024-09-18T00:33:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:33:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15 2024-09-18T00:33:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22 2024-09-18T00:33:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:33:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:33:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2631680.0,"last_snapshot_bytes":5120000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619412,"remote_snapshot_timestamp":1726619412,"replay_state":"idle"} 2024-09-18T00:33:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:17' 2024-09-18T00:33:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-18T00:33:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:33:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-18T00:33:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:33:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-18 00:33:22' 2024-09-18T00:33:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:33:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:22' 2024-09-18T00:33:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1591296.0,"last_snapshot_bytes":3022848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619404,"remote_snapshot_timestamp":1726619404,"replay_state":"idle"}2024-09-18 00:33:17image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619413,"remote_snapshot_timestamp":1726619413,"replay_state":"idle"}2024-09-18 00:33:17image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3704832.0,"last_snapshot_bytes":675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619407,"remote_snapshot_timestamp":1726619407,"replay_state":"idle"}2024-09-18 00:33:17image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3639296.0,"last_snapshot_bytes":3887104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619409,"remote_snapshot_timestamp":1726619409,"replay_state":"idle"}2024-09-18 00:33:17image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stopping_replayremote image demoted2024-09-18 00:33:24image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2631680.0,"last_snapshot_bytes":5120000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619412,"remote_snapshot_timestamp":1726619412,"replay_state":"idle"}2024-09-18 00:33:17' 2024-09-18T00:33:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:33:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.619 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:33:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-18T00:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:24.642 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:33:24.640+0000 7f9bfab49640 -1 reset not still connected to 0x55cd46dbeb60 2024-09-18T00:33:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1591296.0,"last_snapshot_bytes":3022848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619404,"remote_snapshot_timestamp":1726619404,"replay_state":"idle"}2024-09-18 00:33:17image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619413,"remote_snapshot_timestamp":1726619413,"replay_state":"idle"}2024-09-18 00:33:17image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3704832.0,"last_snapshot_bytes":675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619407,"remote_snapshot_timestamp":1726619407,"replay_state":"idle"}2024-09-18 00:33:17image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3639296.0,"last_snapshot_bytes":3887104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619409,"remote_snapshot_timestamp":1726619409,"replay_state":"idle"}2024-09-18 00:33:17image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2631680.0,"last_snapshot_bytes":5120000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619412,"remote_snapshot_timestamp":1726619412,"replay_state":"idle"}2024-09-18 00:33:17' 2024-09-18T00:33:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:33:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1591296.0,"last_snapshot_bytes":3022848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619404,"remote_snapshot_timestamp":1726619404,"replay_state":"idle"}2024-09-18 00:33:17image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619413,"remote_snapshot_timestamp":1726619413,"replay_state":"idle"}2024-09-18 00:33:17image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3704832.0,"last_snapshot_bytes":675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619407,"remote_snapshot_timestamp":1726619407,"replay_state":"idle"}2024-09-18 00:33:17image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3639296.0,"last_snapshot_bytes":3887104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619409,"remote_snapshot_timestamp":1726619409,"replay_state":"idle"}2024-09-18 00:33:17image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2631680.0,"last_snapshot_bytes":5120000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619412,"remote_snapshot_timestamp":1726619412,"replay_state":"idle"}2024-09-18 00:33:17' 2024-09-18T00:33:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:33:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:33:24.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:33:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:24.677 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:24.684 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-18T00:33:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:33:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:24.687 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-18T00:33:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:33:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1591296.0,"last_snapshot_bytes":3022848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619404,"remote_snapshot_timestamp":1726619404,"replay_state":"idle"}2024-09-18 00:33:17image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619413,"remote_snapshot_timestamp":1726619413,"replay_state":"idle"}2024-09-18 00:33:17image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3704832.0,"last_snapshot_bytes":675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619407,"remote_snapshot_timestamp":1726619407,"replay_state":"idle"}2024-09-18 00:33:17image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3639296.0,"last_snapshot_bytes":3887104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619409,"remote_snapshot_timestamp":1726619409,"replay_state":"idle"}2024-09-18 00:33:17image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2631680.0,"last_snapshot_bytes":5120000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619412,"remote_snapshot_timestamp":1726619412,"replay_state":"idle"}2024-09-18 00:33:17' 2024-09-18T00:33:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:24.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:33:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:33:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-18T00:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:33:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: dfd928bc-d98b-418a-8961-dfb82037e5b1 2024-09-18T00:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22 2024-09-18T00:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22' 2024-09-18T00:33:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:24.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:33:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-18 00:33:22' 2024-09-18T00:33:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:33:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:22' 2024-09-18T00:33:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:24.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1591296.0,"last_snapshot_bytes":3022848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619404,"remote_snapshot_timestamp":1726619404,"replay_state":"idle"}2024-09-18 00:33:17image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619413,"remote_snapshot_timestamp":1726619413,"replay_state":"idle"}2024-09-18 00:33:17image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3704832.0,"last_snapshot_bytes":675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619407,"remote_snapshot_timestamp":1726619407,"replay_state":"idle"}2024-09-18 00:33:17image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3639296.0,"last_snapshot_bytes":3887104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619409,"remote_snapshot_timestamp":1726619409,"replay_state":"idle"}2024-09-18 00:33:17image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2631680.0,"last_snapshot_bytes":5120000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619412,"remote_snapshot_timestamp":1726619412,"replay_state":"idle"}2024-09-18 00:33:17' 2024-09-18T00:33:24.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:33:24.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:33:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-18T00:33:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:33:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1591296.0,"last_snapshot_bytes":3022848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619404,"remote_snapshot_timestamp":1726619404,"replay_state":"idle"}2024-09-18 00:33:17image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619413,"remote_snapshot_timestamp":1726619413,"replay_state":"idle"}2024-09-18 00:33:17image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3704832.0,"last_snapshot_bytes":675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619407,"remote_snapshot_timestamp":1726619407,"replay_state":"idle"}2024-09-18 00:33:17image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3639296.0,"last_snapshot_bytes":3887104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619409,"remote_snapshot_timestamp":1726619409,"replay_state":"idle"}2024-09-18 00:33:17image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2631680.0,"last_snapshot_bytes":5120000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619412,"remote_snapshot_timestamp":1726619412,"replay_state":"idle"}2024-09-18 00:33:17' 2024-09-18T00:33:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:33:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:33:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-18T00:33:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:33:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:24.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1591296.0,"last_snapshot_bytes":3022848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619404,"remote_snapshot_timestamp":1726619404,"replay_state":"idle"}2024-09-18 00:33:17image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619413,"remote_snapshot_timestamp":1726619413,"replay_state":"idle"}2024-09-18 00:33:17image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3704832.0,"last_snapshot_bytes":675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619407,"remote_snapshot_timestamp":1726619407,"replay_state":"idle"}2024-09-18 00:33:17image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3639296.0,"last_snapshot_bytes":3887104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619409,"remote_snapshot_timestamp":1726619409,"replay_state":"idle"}2024-09-18 00:33:17image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2631680.0,"last_snapshot_bytes":5120000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619412,"remote_snapshot_timestamp":1726619412,"replay_state":"idle"}2024-09-18 00:33:17' 2024-09-18T00:33:24.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:24.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:33:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:33:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:24.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:24.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-18T00:33:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:33:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1591296.0,"last_snapshot_bytes":3022848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619404,"remote_snapshot_timestamp":1726619404,"replay_state":"idle"}2024-09-18 00:33:17image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619413,"remote_snapshot_timestamp":1726619413,"replay_state":"idle"}2024-09-18 00:33:17image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3704832.0,"last_snapshot_bytes":675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619407,"remote_snapshot_timestamp":1726619407,"replay_state":"idle"}2024-09-18 00:33:17image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3639296.0,"last_snapshot_bytes":3887104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619409,"remote_snapshot_timestamp":1726619409,"replay_state":"idle"}2024-09-18 00:33:17image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2631680.0,"last_snapshot_bytes":5120000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619412,"remote_snapshot_timestamp":1726619412,"replay_state":"idle"}2024-09-18 00:33:17' 2024-09-18T00:33:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:33:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:33:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-18T00:33:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:33:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:24.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1591296.0,"last_snapshot_bytes":3022848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619404,"remote_snapshot_timestamp":1726619404,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:33:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619413,"remote_snapshot_timestamp":1726619413,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:33:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3704832.0,"last_snapshot_bytes":675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619407,"remote_snapshot_timestamp":1726619407,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:33:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3639296.0,"last_snapshot_bytes":3887104,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619409,"remote_snapshot_timestamp":1726619409,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:33:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:22cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2631680.0,"last_snapshot_bytes":5120000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619412,"remote_snapshot_timestamp":1726619412,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:33:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22' 2024-09-18T00:33:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:33:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:33:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-18T00:33:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:33:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:33:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-18T00:33:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1591296.0,"last_snapshot_bytes":3022848,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619404,"remote_snapshot_timestamp":1726619404,"replay_state":"idle"}2024-09-18 00:33:17image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1679360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619413,"remote_snapshot_timestamp":1726619413,"replay_state":"idle"}2024-09-18 00:33:17image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3704832.0,"last_snapshot_bytes":675840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619407,"remote_snapshot_timestamp":1726619407,"replay_state":"idle"}2024-09-18 00:33:17image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stopping_replayremote image demoted2024-09-18 00:33:24image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:24image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:33:22cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stopping_replayremote image demoted2024-09-18 00:33:24' 2024-09-18T00:33:24.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:24.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:33:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:22' 2024-09-18T00:33:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:33:24.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-18T00:33:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:33:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:33:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-18T00:33:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:33:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.640 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-18T00:33:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-18T00:33:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:33:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-18T00:33:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:33:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 3f8781e9-128d-405e-83d4-3ab3eade1052 2024-09-18T00:33:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:33:25.690 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:24 2024-09-18T00:33:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:33:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22' 2024-09-18T00:33:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:33:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-18T00:33:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:33:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-18T00:33:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:25.695 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-18 00:33:24' 2024-09-18T00:33:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:33:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:24' 2024-09-18T00:33:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-18T00:33:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:33:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.707 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-18T00:33:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:33:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 6c655549-4e7a-426d-9a67-53d1811be39f 2024-09-18T00:33:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:33:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:24 2024-09-18T00:33:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:25.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:33:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22' 2024-09-18T00:33:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-18T00:33:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:33:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-18T00:33:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:33:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:25.756 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-18 00:33:24' 2024-09-18T00:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:24' 2024-09-18T00:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-18T00:33:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:33:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-18T00:33:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:33:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ba910667-b6c2-40e5-972a-77379dacdcac 2024-09-18T00:33:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:33:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:24 2024-09-18T00:33:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:33:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22' 2024-09-18T00:33:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:33:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:25.792 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-18 00:33:24' 2024-09-18T00:33:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:33:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:24' 2024-09-18T00:33:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-18T00:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-18T00:33:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:22cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22' 2024-09-18T00:33:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:33:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-18T00:33:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:33:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-18T00:33:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-18T00:33:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:33:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:33:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-18T00:33:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:33:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 688ff7b2-7bdc-4e47-87f8-0ebc17160572 2024-09-18T00:33:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:33:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:24 2024-09-18T00:33:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:33:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22' 2024-09-18T00:33:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:33:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-18 00:33:24' 2024-09-18T00:33:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:33:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:24' 2024-09-18T00:33:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:33:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1eba87df-00c0-43f2-9499-35f1b3ee6e5f 2024-09-18T00:33:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:33:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:24 2024-09-18T00:33:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:33:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22' 2024-09-18T00:33:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:33:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:25.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:25.891 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:25.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-18 00:33:24' 2024-09-18T00:33:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:33:25.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:24' 2024-09-18T00:33:25.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:25.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:33:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-18T00:33:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:33:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:25.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-18T00:33:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:33:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:33:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:33:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a8f733ff-3d00-42ab-a510-55f764b8b3ae 2024-09-18T00:33:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:33:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:24 2024-09-18T00:33:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:33:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22' 2024-09-18T00:33:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-18T00:33:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:22cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22' 2024-09-18T00:33:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:33:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:33:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15 2024-09-18T00:33:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:33:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:24 2024-09-18T00:33:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:33:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22' 2024-09-18T00:33:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:33:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:33:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:33:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-18T00:33:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:33:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:33:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-18T00:33:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-18 00:33:24' 2024-09-18T00:33:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:33:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:24' 2024-09-18T00:33:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-18 00:33:24' 2024-09-18T00:33:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:33:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:24' 2024-09-18T00:33:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:22cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22' 2024-09-18T00:33:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:33:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9595931d-b126-4a67-bfbd-d2ac477b0311 2024-09-18T00:33:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:33:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:24 2024-09-18T00:33:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:33:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22' 2024-09-18T00:33:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:33:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:33:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:33:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:26.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-18T00:33:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:33:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:33:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-18T00:33:26.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-18 00:33:24' 2024-09-18T00:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:24' 2024-09-18T00:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:26.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:22cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22' 2024-09-18T00:33:26.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:33:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:33:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:33:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 6dd62243-5126-440f-919f-f75472281ac7 2024-09-18T00:33:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:26.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:33:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:33:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:24 2024-09-18T00:33:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:33:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:33:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:33:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:33:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:22' 2024-09-18T00:33:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:33:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:33:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:33:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:33:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:33:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:33:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-18T00:33:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:33:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:33:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-18T00:33:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:33:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:22cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22' 2024-09-18T00:33:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:26.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:33:26.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:33:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:33:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:33:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-18 00:33:24' 2024-09-18T00:33:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:33:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:33:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:24' 2024-09-18T00:33:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:33:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:33:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:33:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:33:26.181 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd0 2024-09-18T00:33:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:26.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:33:26.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-18T00:33:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:26.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-18T00:33:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:33:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:33:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-18T00:33:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:22cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22' 2024-09-18T00:33:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:33:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:33:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:26.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-18T00:33:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:33:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:33:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-18T00:33:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:22cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22' 2024-09-18T00:33:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:22cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22' 2024-09-18T00:33:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:33:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:33:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:33:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:33:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-18T00:33:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:33:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:33:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-18T00:33:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-18T00:33:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:33:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:33:26.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-18T00:33:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1042604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary66dd62243-5126-440f-919f-f75472281ac7up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:22cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:33:24cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:33:22' 2024-09-18T00:33:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:33:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:33:26.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:24' 2024-09-18T00:33:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:33:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:33:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:33:26.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:33:26.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:33:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:33:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-18T00:33:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:33:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:33:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:33:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-18T00:33:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:33:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:33:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:33:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:33:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:33:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:33:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:33:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:33:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:33:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:33:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:33:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-18T00:33:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-18T00:33:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:33:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:33:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:33:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-18T00:33:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:33:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:33:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:33:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-18T00:33:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:33:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:33:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:33:26.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:33:26.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-18T00:33:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-18T00:33:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:33:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:33:26.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-18T00:33:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:33:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:33:26.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-18T00:33:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:33:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:33:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-18T00:33:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd1 2024-09-18T00:33:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-18T00:33:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:33:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd3 2024-09-18T00:33:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-18T00:33:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:33:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd2 2024-09-18T00:33:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-18T00:33:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:33:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd7 2024-09-18T00:33:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-18T00:33:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd4 2024-09-18T00:33:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:33:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-18T00:33:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:33:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd8 2024-09-18T00:33:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:33:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd6 2024-09-18T00:33:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd5 2024-09-18T00:33:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-18T00:33:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd9 2024-09-18T00:33:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-18T00:33:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-18T00:33:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-18T00:33:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:33:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:33:27.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:33:29.405 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:33:29.403+0000 7fd413089640 -1 reset not still connected to 0x5604a9af0b60 2024-09-18T00:33:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=c1c758bc9322e346c3489ebb0a36c5ec 2024-09-18T00:33:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-18T00:33:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ c1c758bc9322e346c3489ebb0a36c5ec != \c\1\c\7\5\8\b\c\9\3\2\2\e\3\4\6\c\3\4\8\9\e\b\b\0\a\3\6\c\5\e\c ]] 2024-09-18T00:33:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=555a93fbc64edf9288910e7a86060018 2024-09-18T00:33:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-18T00:33:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 555a93fbc64edf9288910e7a86060018 != \5\5\5\a\9\3\f\b\c\6\4\e\d\f\9\2\8\8\9\1\0\e\7\a\8\6\0\6\0\0\1\8 ]] 2024-09-18T00:33:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=bd980d751b9a15c8decdf4fc50bd64f6 2024-09-18T00:33:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-18T00:33:58.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=395b4d5d06656c3b34a21075aaa327a8 2024-09-18T00:33:58.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-18T00:33:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ bd980d751b9a15c8decdf4fc50bd64f6 != \b\d\9\8\0\d\7\5\1\b\9\a\1\5\c\8\d\e\c\d\f\4\f\c\5\0\b\d\6\4\f\6 ]] 2024-09-18T00:33:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=3dbd85a9bd05a06145e3f4f2fad165b6 2024-09-18T00:33:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-18T00:33:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 395b4d5d06656c3b34a21075aaa327a8 != \3\9\5\b\4\d\5\d\0\6\6\5\6\c\3\b\3\4\a\2\1\0\7\5\a\a\a\3\2\7\a\8 ]] 2024-09-18T00:33:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 3dbd85a9bd05a06145e3f4f2fad165b6 != \3\d\b\d\8\5\a\9\b\d\0\5\a\0\6\1\4\5\e\3\f\4\f\2\f\a\d\1\6\5\b\6 ]] 2024-09-18T00:33:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=1e573f91c70a1dfdddd1118a7f8e7547 2024-09-18T00:33:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-18T00:33:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 1e573f91c70a1dfdddd1118a7f8e7547 != \1\e\5\7\3\f\9\1\c\7\0\a\1\d\f\d\d\d\d\1\1\1\8\a\7\f\8\e\7\5\4\7 ]] 2024-09-18T00:33:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:33:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 138502 2024-09-18T00:33:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=16e6707b68e924fc529bd4cfe118229c 2024-09-18T00:33:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-18T00:33:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 16e6707b68e924fc529bd4cfe118229c != \1\6\e\6\7\0\7\b\6\8\e\9\2\4\f\c\5\2\9\b\d\4\c\f\e\1\1\8\2\2\9\c ]] 2024-09-18T00:33:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=1a7a3ebad130cc0996c97ffb3e6bdbc0 2024-09-18T00:33:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-18T00:33:59.796 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 1a7a3ebad130cc0996c97ffb3e6bdbc0 != \1\a\7\a\3\e\b\a\d\1\3\0\c\c\0\9\9\6\c\9\7\f\f\b\3\e\6\b\d\b\c\0 ]] 2024-09-18T00:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 138504 2024-09-18T00:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 138506 2024-09-18T00:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:33:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 138508 2024-09-18T00:33:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=bfd99bccb242d7e858c15a109dd3d5e2 2024-09-18T00:33:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-18T00:33:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ bfd99bccb242d7e858c15a109dd3d5e2 != \b\f\d\9\9\b\c\c\b\2\4\2\d\7\e\8\5\8\c\1\5\a\1\0\9\d\d\3\d\5\e\2 ]] 2024-09-18T00:33:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=f93a51c035317657007be9fc31fa160d 2024-09-18T00:33:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-18T00:33:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ f93a51c035317657007be9fc31fa160d != \f\9\3\a\5\1\c\0\3\5\3\1\7\6\5\7\0\0\7\b\e\9\f\c\3\1\f\a\1\6\0\d ]] 2024-09-18T00:33:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:33:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 138510 2024-09-18T00:33:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:33:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 138512 2024-09-18T00:33:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:33:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 138514 2024-09-18T00:33:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:33:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 138516 2024-09-18T00:33:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:33:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 138517 2024-09-18T00:33:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:33:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:33:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-18T00:33:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-18T00:33:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:33:59.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:33:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-18T00:33:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:33:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:33:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:33:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:33:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:33:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:33:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:33:59.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:34:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-18T00:34:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:34:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:34:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:34:01.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ba910667-b6c2-40e5-972a-77379dacdcac 2024-09-18T00:34:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:34:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":6540492.8,"bytes_per_snapshot":65404928.0,"last_snapshot_bytes":65404928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:34:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:47 2024-09-18T00:34:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:34:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:34:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:34:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:34:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:57' 2024-09-18T00:34:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:34:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:34:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:34:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":6540492.8,"bytes_per_snapshot":65404928.0,"last_snapshot_bytes":65404928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:34:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:34:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:34:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:34:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:34:01.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:34:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:47' 2024-09-18T00:34:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-18 00:33:47' 2024-09-18T00:34:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:34:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:47' 2024-09-18T00:34:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:01.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:34:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary1ba910667-b6c2-40e5-972a-77379dacdcacup+replayingreplaying, {"bytes_per_second":6540492.8,"bytes_per_snapshot":65404928.0,"last_snapshot_bytes":65404928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:33:57image-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+replayingreplaying, {"bytes_per_second":7178649.6,"bytes_per_snapshot":71786496.0,"last_snapshot_bytes":71786496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:33:57image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+replayingreplaying, {"bytes_per_second":6366003.2,"bytes_per_snapshot":63660032.0,"last_snapshot_bytes":63660032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:33:57image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+replayingreplaying, {"bytes_per_second":6264422.4,"bytes_per_snapshot":62644224.0,"last_snapshot_bytes":62644224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:33:57image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+replayingreplaying, {"bytes_per_second":6282444.8,"bytes_per_snapshot":62824448.0,"last_snapshot_bytes":62824448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:33:57image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+replayingreplaying, {"bytes_per_second":6284902.4,"bytes_per_snapshot":62849024.0,"last_snapshot_bytes":62849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:33:57image-primary66dd62243-5126-440f-919f-f75472281ac7up+replayingreplaying, {"bytes_per_second":6313984.0,"bytes_per_snapshot":63139840.0,"last_snapshot_bytes":63139840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:33:57image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+replayingreplaying, {"bytes_per_second":6648217.6,"bytes_per_snapshot":66482176.0,"last_snapshot_bytes":66482176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:33:57image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+replayingreplaying, {"bytes_per_second":6533939.2,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619605,"remote_snapshot_timestamp":1726619605,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:33:57image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+replayingreplaying, {"bytes_per_second":6112870.4,"bytes_per_snapshot":61128704.0,"last_snapshot_bytes":61128704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:33:57' 2024-09-18T00:34:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:34:01.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:34:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:47' 2024-09-18T00:34:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:34:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:34:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:34:01.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-18T00:34:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:34:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:34:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-18T00:34:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-18T00:34:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:34:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-18T00:34:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:34:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:34:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary1 2024-09-18T00:34:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary1 2024-09-18T00:34:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-18T00:34:03.237 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-18T00:34:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:34:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-18T00:34:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:34:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-18T00:34:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:34:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:34:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a8f733ff-3d00-42ab-a510-55f764b8b3ae 2024-09-18T00:34:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:34:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":6366003.2,"bytes_per_snapshot":63660032.0,"last_snapshot_bytes":63660032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:34:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:47 2024-09-18T00:34:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:34:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:34:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:34:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:34:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:34:02' 2024-09-18T00:34:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:34:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:34:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:34:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":6366003.2,"bytes_per_snapshot":63660032.0,"last_snapshot_bytes":63660032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:34:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:34:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:34:04.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:34:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:34:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:34:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:47' 2024-09-18T00:34:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-18 00:33:47' 2024-09-18T00:34:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:47' 2024-09-18T00:34:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:04.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:34:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK942634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+replayingreplaying, {"bytes_per_second":7178649.6,"bytes_per_snapshot":71786496.0,"last_snapshot_bytes":71786496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02image-primary2a8f733ff-3d00-42ab-a510-55f764b8b3aeup+replayingreplaying, {"bytes_per_second":6366003.2,"bytes_per_snapshot":63660032.0,"last_snapshot_bytes":63660032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+replayingreplaying, {"bytes_per_second":6264422.4,"bytes_per_snapshot":62644224.0,"last_snapshot_bytes":62644224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+replayingreplaying, {"bytes_per_second":6282444.8,"bytes_per_snapshot":62824448.0,"last_snapshot_bytes":62824448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+replayingreplaying, {"bytes_per_second":6284902.4,"bytes_per_snapshot":62849024.0,"last_snapshot_bytes":62849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02image-primary66dd62243-5126-440f-919f-f75472281ac7up+replayingreplaying, {"bytes_per_second":6313984.0,"bytes_per_snapshot":63139840.0,"last_snapshot_bytes":63139840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+replayingreplaying, {"bytes_per_second":6648217.6,"bytes_per_snapshot":66482176.0,"last_snapshot_bytes":66482176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+replayingreplaying, {"bytes_per_second":6533939.2,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619605,"remote_snapshot_timestamp":1726619605,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+replayingreplaying, {"bytes_per_second":6112870.4,"bytes_per_snapshot":61128704.0,"last_snapshot_bytes":61128704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02' 2024-09-18T00:34:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:34:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:34:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:47' 2024-09-18T00:34:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:34:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:34:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:04.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:34:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-18T00:34:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:34:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-18T00:34:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-18T00:34:05.369 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:34:05.367+0000 7f706e974640 -1 reset not still connected to 0x55df30d688f0 2024-09-18T00:34:05.369 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:34:05.367+0000 7f706e974640 -1 reset not still connected to 0x55df326c3860 2024-09-18T00:34:05.370 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:34:05.369+0000 7f706e974640 -1 reset not still connected to 0x55df38540680 2024-09-18T00:34:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:34:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-18T00:34:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:34:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary2 2024-09-18T00:34:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary2 2024-09-18T00:34:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-18T00:34:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-18T00:34:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:34:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-18T00:34:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:34:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-18T00:34:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:34:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:34:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 6c655549-4e7a-426d-9a67-53d1811be39f 2024-09-18T00:34:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:34:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":6264422.4,"bytes_per_snapshot":62644224.0,"last_snapshot_bytes":62644224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:34:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:47 2024-09-18T00:34:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:34:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:34:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:34:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:34:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:34:02' 2024-09-18T00:34:07.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:34:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:34:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:34:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":6264422.4,"bytes_per_snapshot":62644224.0,"last_snapshot_bytes":62644224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:34:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:34:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:34:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:34:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:34:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:34:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:47' 2024-09-18T00:34:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-18 00:33:47' 2024-09-18T00:34:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:47' 2024-09-18T00:34:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:07.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:34:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK842634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+replayingreplaying, {"bytes_per_second":7178649.6,"bytes_per_snapshot":71786496.0,"last_snapshot_bytes":71786496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02image-primary36c655549-4e7a-426d-9a67-53d1811be39fup+replayingreplaying, {"bytes_per_second":6264422.4,"bytes_per_snapshot":62644224.0,"last_snapshot_bytes":62644224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+replayingreplaying, {"bytes_per_second":6282444.8,"bytes_per_snapshot":62824448.0,"last_snapshot_bytes":62824448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+replayingreplaying, {"bytes_per_second":6284902.4,"bytes_per_snapshot":62849024.0,"last_snapshot_bytes":62849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02image-primary66dd62243-5126-440f-919f-f75472281ac7up+replayingreplaying, {"bytes_per_second":6313984.0,"bytes_per_snapshot":63139840.0,"last_snapshot_bytes":63139840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+replayingreplaying, {"bytes_per_second":6648217.6,"bytes_per_snapshot":66482176.0,"last_snapshot_bytes":66482176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+replayingreplaying, {"bytes_per_second":6533939.2,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619605,"remote_snapshot_timestamp":1726619605,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+replayingreplaying, {"bytes_per_second":6112870.4,"bytes_per_snapshot":61128704.0,"last_snapshot_bytes":61128704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:02' 2024-09-18T00:34:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:34:07.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:34:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:47' 2024-09-18T00:34:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:34:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:34:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:34:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-18T00:34:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:34:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-18T00:34:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-18T00:34:09.201 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:34:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-18T00:34:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:34:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary3 2024-09-18T00:34:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary3 2024-09-18T00:34:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-18T00:34:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-18T00:34:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:34:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-18T00:34:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:34:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-18T00:34:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-18T00:34:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:34:10.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1eba87df-00c0-43f2-9499-35f1b3ee6e5f 2024-09-18T00:34:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:34:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":6282444.8,"bytes_per_snapshot":62824448.0,"last_snapshot_bytes":62824448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:34:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:47 2024-09-18T00:34:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:34:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:34:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:34:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:34:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:34:07' 2024-09-18T00:34:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:34:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:34:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:34:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":6282444.8,"bytes_per_snapshot":62824448.0,"last_snapshot_bytes":62824448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:34:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:34:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:34:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:34:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:34:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:34:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:47' 2024-09-18T00:34:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-18 00:33:47' 2024-09-18T00:34:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:47' 2024-09-18T00:34:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:10.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:34:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK742634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+replayingreplaying, {"bytes_per_second":7178649.6,"bytes_per_snapshot":71786496.0,"last_snapshot_bytes":71786496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:07image-primary41eba87df-00c0-43f2-9499-35f1b3ee6e5fup+replayingreplaying, {"bytes_per_second":6282444.8,"bytes_per_snapshot":62824448.0,"last_snapshot_bytes":62824448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:07image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+replayingreplaying, {"bytes_per_second":6284902.4,"bytes_per_snapshot":62849024.0,"last_snapshot_bytes":62849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:07image-primary66dd62243-5126-440f-919f-f75472281ac7up+replayingreplaying, {"bytes_per_second":6313984.0,"bytes_per_snapshot":63139840.0,"last_snapshot_bytes":63139840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:07image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+replayingreplaying, {"bytes_per_second":6648217.6,"bytes_per_snapshot":66482176.0,"last_snapshot_bytes":66482176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:07image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+replayingreplaying, {"bytes_per_second":6533939.2,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619605,"remote_snapshot_timestamp":1726619605,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:07image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+replayingreplaying, {"bytes_per_second":6112870.4,"bytes_per_snapshot":61128704.0,"last_snapshot_bytes":61128704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:07' 2024-09-18T00:34:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:34:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:34:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:47' 2024-09-18T00:34:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:34:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:34:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:34:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-18T00:34:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:34:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-18T00:34:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-18T00:34:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:34:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-18T00:34:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:34:12.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary4 2024-09-18T00:34:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:12.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary4 2024-09-18T00:34:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-18T00:34:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-18T00:34:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:34:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-18T00:34:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:34:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-18T00:34:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-18T00:34:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:34:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9595931d-b126-4a67-bfbd-d2ac477b0311 2024-09-18T00:34:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:34:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":6284902.4,"bytes_per_snapshot":62849024.0,"last_snapshot_bytes":62849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:34:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:47 2024-09-18T00:34:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:34:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:34:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:34:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:34:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:34:12' 2024-09-18T00:34:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:34:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:34:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:34:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":6284902.4,"bytes_per_snapshot":62849024.0,"last_snapshot_bytes":62849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:34:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:13.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:34:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:34:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:34:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:34:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:34:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:47' 2024-09-18T00:34:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-18 00:33:47' 2024-09-18T00:34:13.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:47' 2024-09-18T00:34:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:34:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK642634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+replayingreplaying, {"bytes_per_second":7178649.6,"bytes_per_snapshot":71786496.0,"last_snapshot_bytes":71786496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:12image-primary59595931d-b126-4a67-bfbd-d2ac477b0311up+replayingreplaying, {"bytes_per_second":6284902.4,"bytes_per_snapshot":62849024.0,"last_snapshot_bytes":62849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:12image-primary66dd62243-5126-440f-919f-f75472281ac7up+replayingreplaying, {"bytes_per_second":6313984.0,"bytes_per_snapshot":63139840.0,"last_snapshot_bytes":63139840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:12image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+replayingreplaying, {"bytes_per_second":6648217.6,"bytes_per_snapshot":66482176.0,"last_snapshot_bytes":66482176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:12image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+replayingreplaying, {"bytes_per_second":6533939.2,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619605,"remote_snapshot_timestamp":1726619605,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:12image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+replayingreplaying, {"bytes_per_second":6112870.4,"bytes_per_snapshot":61128704.0,"last_snapshot_bytes":61128704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:12' 2024-09-18T00:34:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:34:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:34:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:47' 2024-09-18T00:34:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:34:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:34:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:34:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-18T00:34:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:34:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-18T00:34:13.463 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-18T00:34:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:34:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-18T00:34:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:34:15.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary5 2024-09-18T00:34:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary5 2024-09-18T00:34:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-18T00:34:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-18T00:34:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:34:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-18T00:34:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:34:15.272 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:34:15.267+0000 7f706e974640 -1 reset not still connected to 0x55df30d688f0 2024-09-18T00:34:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-18T00:34:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:34:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:34:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 6dd62243-5126-440f-919f-f75472281ac7 2024-09-18T00:34:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:34:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":6313984.0,"bytes_per_snapshot":63139840.0,"last_snapshot_bytes":63139840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:34:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:33:47 2024-09-18T00:34:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:34:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:34:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:34:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:34:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:34:12' 2024-09-18T00:34:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:34:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:34:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:34:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":6313984.0,"bytes_per_snapshot":63139840.0,"last_snapshot_bytes":63139840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:34:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:34:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:34:16.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:34:16.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:34:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:34:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:47' 2024-09-18T00:34:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-18 00:33:47' 2024-09-18T00:34:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:33:47' 2024-09-18T00:34:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:34:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK542634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+replayingreplaying, {"bytes_per_second":7178649.6,"bytes_per_snapshot":71786496.0,"last_snapshot_bytes":71786496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:12image-primary66dd62243-5126-440f-919f-f75472281ac7up+replayingreplaying, {"bytes_per_second":6313984.0,"bytes_per_snapshot":63139840.0,"last_snapshot_bytes":63139840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:12image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+replayingreplaying, {"bytes_per_second":6648217.6,"bytes_per_snapshot":66482176.0,"last_snapshot_bytes":66482176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:12image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+replayingreplaying, {"bytes_per_second":6533939.2,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619605,"remote_snapshot_timestamp":1726619605,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:12image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+replayingreplaying, {"bytes_per_second":6112870.4,"bytes_per_snapshot":61128704.0,"last_snapshot_bytes":61128704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:33:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:12' 2024-09-18T00:34:16.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:34:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:34:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:33:47' 2024-09-18T00:34:16.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:34:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:34:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:34:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-18T00:34:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:34:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-18T00:34:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-18T00:34:17.991 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:17.989+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782eea0 2024-09-18T00:34:17.991 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:17.989+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-18T00:34:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:34:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-18T00:34:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:34:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary6 2024-09-18T00:34:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary6 2024-09-18T00:34:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-18T00:34:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-18T00:34:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:34:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:34:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-18T00:34:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:34:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:34:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-18T00:34:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:34:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:19.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:34:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:34:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 688ff7b2-7bdc-4e47-87f8-0ebc17160572 2024-09-18T00:34:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:34:19.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66482176.0,"last_snapshot_bytes":66482176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:34:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:34:17 2024-09-18T00:34:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:34:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:34:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:34:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:34:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:34:17' 2024-09-18T00:34:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:34:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:34:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:34:19.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66482176.0,"last_snapshot_bytes":66482176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:34:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:34:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:34:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:34:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:34:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:34:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:34:17' 2024-09-18T00:34:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-18 00:34:17' 2024-09-18T00:34:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:34:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:34:17' 2024-09-18T00:34:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:34:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71786496.0,"last_snapshot_bytes":71786496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:34:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:17image-primary7688ff7b2-7bdc-4e47-87f8-0ebc17160572up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":66482176.0,"last_snapshot_bytes":66482176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:34:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:17image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619605,"remote_snapshot_timestamp":1726619605,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:34:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:17image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61128704.0,"last_snapshot_bytes":61128704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:34:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:17' 2024-09-18T00:34:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:34:19.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:34:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:34:17' 2024-09-18T00:34:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:34:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:34:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:34:19.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-18T00:34:19.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:34:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:34:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-18T00:34:19.500 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-18T00:34:19.862 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:19.861+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-18T00:34:20.304 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:34:20.300+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:34:20.304 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:34:20.300+0000 7f706e974640 -1 reset not still connected to 0x55df30d688f0 2024-09-18T00:34:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:34:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-18T00:34:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:34:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:34:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary7 2024-09-18T00:34:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary7 2024-09-18T00:34:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-18T00:34:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-18T00:34:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:34:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:34:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-18T00:34:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:34:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:34:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-18T00:34:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:34:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:34:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:34:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: dfd928bc-d98b-418a-8961-dfb82037e5b1 2024-09-18T00:34:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:34:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619605,"remote_snapshot_timestamp":1726619605,"replay_state":"idle"} 2024-09-18T00:34:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:34:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:34:17 2024-09-18T00:34:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:34:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:34:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:34:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:34:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:34:17' 2024-09-18T00:34:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:34:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:34:22.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:34:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619605,"remote_snapshot_timestamp":1726619605,"replay_state":"idle"} 2024-09-18T00:34:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:34:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:34:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:22.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:34:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:34:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:34:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:34:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:34:17' 2024-09-18T00:34:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-18 00:34:17' 2024-09-18T00:34:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:34:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:34:17' 2024-09-18T00:34:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:22.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:22.315 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:34:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK342634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71786496.0,"last_snapshot_bytes":71786496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:34:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:17image-primary8dfd928bc-d98b-418a-8961-dfb82037e5b1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65339392.0,"last_snapshot_bytes":65339392,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619605,"remote_snapshot_timestamp":1726619605,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:34:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:17image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61128704.0,"last_snapshot_bytes":61128704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:34:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:17' 2024-09-18T00:34:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:34:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:34:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:34:17' 2024-09-18T00:34:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:34:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:34:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:34:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-18T00:34:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:34:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:34:22.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-18T00:34:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-18T00:34:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:34:24.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-18T00:34:24.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:24.194 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:34:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:34:24.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary8 2024-09-18T00:34:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:24.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary8 2024-09-18T00:34:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-18T00:34:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-18T00:34:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:34:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:34:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-18T00:34:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:34:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:24.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:24.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:34:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-18T00:34:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:34:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:25.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:34:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:34:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15 2024-09-18T00:34:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:34:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61128704.0,"last_snapshot_bytes":61128704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:34:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:34:17 2024-09-18T00:34:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:34:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:34:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:34:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:34:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:34:22' 2024-09-18T00:34:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:34:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:34:25.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:34:25.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61128704.0,"last_snapshot_bytes":61128704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:25.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:34:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:34:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:34:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:34:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:34:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:34:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:34:17' 2024-09-18T00:34:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-18 00:34:17' 2024-09-18T00:34:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:34:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:34:17' 2024-09-18T00:34:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:34:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK242634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71786496.0,"last_snapshot_bytes":71786496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:34:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:22image-primary9ec1ed005-fd2c-4a25-89cf-2c1b4b6cda15up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61128704.0,"last_snapshot_bytes":61128704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:34:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:22' 2024-09-18T00:34:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:34:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:34:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:34:17' 2024-09-18T00:34:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:34:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:34:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:34:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-18T00:34:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:34:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:34:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-18T00:34:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-18T00:34:27.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:34:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-18T00:34:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:34:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:34:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary9 2024-09-18T00:34:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary9 2024-09-18T00:34:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-18T00:34:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-18T00:34:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:34:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:34:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-18T00:34:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:34:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:34:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-18T00:34:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:28.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:34:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:34:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:34:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:34:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:34:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 3f8781e9-128d-405e-83d4-3ab3eade1052 2024-09-18T00:34:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:34:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71786496.0,"last_snapshot_bytes":71786496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:34:28.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:34:17 2024-09-18T00:34:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:34:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:34:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:34:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:34:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:34:27' 2024-09-18T00:34:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:34:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:34:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:34:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71786496.0,"last_snapshot_bytes":71786496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"} 2024-09-18T00:34:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:34:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:34:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:34:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:34:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:34:28.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:34:28.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:34:17' 2024-09-18T00:34:28.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-18 00:34:17' 2024-09-18T00:34:28.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:34:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:34:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:34:17' 2024-09-18T00:34:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:34:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:34:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:34:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary103f8781e9-128d-405e-83d4-3ab3eade1052up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":71786496.0,"last_snapshot_bytes":71786496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619606,"remote_snapshot_timestamp":1726619606,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:34:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:34:27' 2024-09-18T00:34:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:34:28.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:34:28.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:34:17' 2024-09-18T00:34:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:34:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:34:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:34:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:34:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:34:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-18T00:34:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:34:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:34:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-18T00:34:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-18T00:34:30.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:34:30.273 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-18T00:34:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:34:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:34:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:34:30.274 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary10 2024-09-18T00:34:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary10 2024-09-18T00:34:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..10} 2024-09-18T00:34:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS=() 2024-09-18T00:34:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS=() 2024-09-18T00:34:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS=() 2024-09-18T00:34:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS=() 2024-09-18T00:34:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ RET=0 2024-09-18T00:34:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-18T00:34:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary1 2024-09-18T00:34:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-18T00:34:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:34:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:34:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-18T00:34:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:34:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:34:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:34:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:34:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-18T00:34:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:34:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:34:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-18T00:34:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:34:30.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:30.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-18T00:34:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:34:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-18T00:34:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary1': 2024-09-18T00:34:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:34:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:34:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:34:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 39785c3b66fc 2024-09-18T00:34:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.39785c3b66fc 2024-09-18T00:34:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:34:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:34:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:34:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:34:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:34:30 2024 2024-09-18T00:34:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:34:30 2024 2024-09-18T00:34:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:34:30 2024 2024-09-18T00:34:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:34:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:34:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 61594334-abd8-44fc-9715-2484fbb2143a 2024-09-18T00:34:31.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:34:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:34:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:34:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary1 2024-09-18T00:34:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd0 2024-09-18T00:34:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:34:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd0 2024-09-18T00:34:31.233 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:34:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:34:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:34:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 87eb4071-826d-4e6c-a9db-8d6ac2c50183 2024-09-18T00:34:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:34:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:34:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:34:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:34:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:34:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:34:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary1 2024-09-18T00:34:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd0 test-primary1 2024-09-18T00:34:31.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:34:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary1 2024-09-18T00:34:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:34:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-18T00:34:31.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary1 2024-09-18T00:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-18T00:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary2 2024-09-18T00:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-18T00:34:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:34:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:34:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary1 2024-09-18T00:34:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-18T00:34:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:34:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-18T00:34:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:34:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:34:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:34:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-18T00:34:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:34:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:34:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:31.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-18T00:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-18T00:34:31.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:34:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-18T00:34:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:34:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2978 2024-09-18T00:34:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-18T00:34:32.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary2': 2024-09-18T00:34:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:34:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:34:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:34:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 3988252223f7 2024-09-18T00:34:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.3988252223f7 2024-09-18T00:34:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:34:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:34:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:34:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:34:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:34:31 2024 2024-09-18T00:34:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:34:31 2024 2024-09-18T00:34:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:34:31 2024 2024-09-18T00:34:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:34:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:34:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 9d6ca525-e53f-4cf8-80fc-35c7b5df9fac 2024-09-18T00:34:32.132 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:34:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:34:32.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:34:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary2 2024-09-18T00:34:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd1 2024-09-18T00:34:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:34:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd1 2024-09-18T00:34:32.214 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:34:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:34:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:34:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: fa8999b3-13a2-4a22-b332-9b21d60fd025 2024-09-18T00:34:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:34:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:34:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:34:32.220 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:34:32.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:34:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:34:32.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary2 2024-09-18T00:34:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd1 test-primary2 2024-09-18T00:34:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:34:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary2 2024-09-18T00:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-18T00:34:32.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary2 2024-09-18T00:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-18T00:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:32.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary3 2024-09-18T00:34:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-18T00:34:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:34:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:32.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary2 2024-09-18T00:34:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-18T00:34:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:34:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-18T00:34:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:34:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:34:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:34:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:32.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-18T00:34:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:34:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:34:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-18T00:34:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-18T00:34:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:34:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-18T00:34:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:34:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2980 2024-09-18T00:34:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-18T00:34:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary3': 2024-09-18T00:34:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:34:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:34:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:34:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 399739e884a9 2024-09-18T00:34:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.399739e884a9 2024-09-18T00:34:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:34:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:34:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:34:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:34:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:34:32 2024 2024-09-18T00:34:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:34:32 2024 2024-09-18T00:34:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:34:32 2024 2024-09-18T00:34:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:34:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:34:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 91db1fb0-ad2a-47fe-bde5-20bf3e90ea37 2024-09-18T00:34:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:34:33.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:34:33.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:34:33.150 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary3 2024-09-18T00:34:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd2 2024-09-18T00:34:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:34:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd2 2024-09-18T00:34:33.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:34:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:34:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:34:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 7e592799-2781-4cdc-9aa3-6649bd3f7b66 2024-09-18T00:34:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:34:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:34:33.234 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:34:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:34:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:34:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:34:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary3 2024-09-18T00:34:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd2 test-primary3 2024-09-18T00:34:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:34:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary3 2024-09-18T00:34:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:34:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-18T00:34:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:33.330 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:34:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:34:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary3 2024-09-18T00:34:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-18T00:34:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary4 2024-09-18T00:34:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-18T00:34:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:33.331 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:34:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary3 2024-09-18T00:34:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-18T00:34:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:34:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-18T00:34:33.332 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:34:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:34:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:34:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:33.333 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-18T00:34:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:34:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:34:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-18T00:34:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:33.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-18T00:34:33.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:34:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-18T00:34:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:34:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2982 2024-09-18T00:34:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-18T00:34:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary4': 2024-09-18T00:34:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:34:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:34:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:34:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 39a65108d6e3 2024-09-18T00:34:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.39a65108d6e3 2024-09-18T00:34:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:34:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:34:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:34:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:34:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:34:33 2024 2024-09-18T00:34:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:34:33 2024 2024-09-18T00:34:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:34:33 2024 2024-09-18T00:34:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:34:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:34:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: a0539cb4-609b-47fd-8ae0-93b0a3b749f0 2024-09-18T00:34:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:34:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:34:34.186 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:34:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary4 2024-09-18T00:34:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd3 2024-09-18T00:34:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:34:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd3 2024-09-18T00:34:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:34:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:34:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:34:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 6a456d5e-4fd4-45ca-9ffe-e81b470c78f9 2024-09-18T00:34:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:34:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:34:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:34:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:34:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:34:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:34:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary4 2024-09-18T00:34:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd3 test-primary4 2024-09-18T00:34:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:34:34.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary4 2024-09-18T00:34:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:34:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-18T00:34:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:34.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:34:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:34:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary4 2024-09-18T00:34:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:34:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-18T00:34:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary5 2024-09-18T00:34:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-18T00:34:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary4 2024-09-18T00:34:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-18T00:34:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:34:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-18T00:34:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:34:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:34:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:34:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-18T00:34:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:34:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:34:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-18T00:34:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-18T00:34:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:34:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-18T00:34:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:34:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2984 2024-09-18T00:34:34.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-18T00:34:34.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary5': 2024-09-18T00:34:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:34:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:34:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:34:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 39b4881ce6d3 2024-09-18T00:34:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.39b4881ce6d3 2024-09-18T00:34:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:34:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:34:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:34:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:34:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:34:34 2024 2024-09-18T00:34:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:34:34 2024 2024-09-18T00:34:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:34:34 2024 2024-09-18T00:34:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:34:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:34:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 6edb0522-635d-46cd-8cf6-76f54f3926a0 2024-09-18T00:34:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:34:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:34:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:34:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary5 2024-09-18T00:34:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:34:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:34:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:34:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd4 2024-09-18T00:34:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:34:35.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd4 2024-09-18T00:34:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:34:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:34:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:34:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 70639545-63c7-437b-ab26-c839e1004097 2024-09-18T00:34:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:34:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:34:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:34:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:34:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:34:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:34:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary5 2024-09-18T00:34:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd4 test-primary5 2024-09-18T00:34:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:34:35.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary5 2024-09-18T00:34:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:34:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-18T00:34:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:34:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:34:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary5 2024-09-18T00:34:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-18T00:34:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary6 2024-09-18T00:34:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-18T00:34:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:34:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary5 2024-09-18T00:34:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-18T00:34:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:34:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-18T00:34:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:34:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:34:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:34:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-18T00:34:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:34:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:34:35.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-18T00:34:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:35.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-18T00:34:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:34:35.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-18T00:34:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:34:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:34:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:34:36.113 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:34:36.110+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb5790 2024-09-18T00:34:36.117 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:34:36.116+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:34:36.117 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:34:36.116+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-18T00:34:36.117 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:34:36.116+0000 7f706e974640 -1 reset not still connected to 0x55df3d71eea0 2024-09-18T00:34:36.128 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2987 2024-09-18T00:34:36.129 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-18T00:34:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary6': 2024-09-18T00:34:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:34:36.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:34:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:34:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 39c4b14b16eb 2024-09-18T00:34:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.39c4b14b16eb 2024-09-18T00:34:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:34:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:34:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:34:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:34:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:34:35 2024 2024-09-18T00:34:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:34:35 2024 2024-09-18T00:34:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:34:35 2024 2024-09-18T00:34:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:34:36.267 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:34:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: c19919df-d5b3-43ec-abfb-adf02bda5de0 2024-09-18T00:34:36.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:34:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:34:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:34:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary6 2024-09-18T00:34:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd5 2024-09-18T00:34:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:34:36.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd5 2024-09-18T00:34:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:34:36.409 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:34:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: eadd0e1d-61f7-4b92-a9ad-f0247347f9dc 2024-09-18T00:34:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:34:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:34:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:34:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:34:36.411 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:34:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:34:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary6 2024-09-18T00:34:36.468 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd5 test-primary6 2024-09-18T00:34:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:34:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary6 2024-09-18T00:34:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:34:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-18T00:34:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:34:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:34:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary6 2024-09-18T00:34:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-18T00:34:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary7 2024-09-18T00:34:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-18T00:34:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:34:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary6 2024-09-18T00:34:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:34:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-18T00:34:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:34:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-18T00:34:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:34:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:34:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:34:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:34:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-18T00:34:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:34:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:34:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-18T00:34:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:34:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-18T00:34:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:34:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-18T00:34:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2986 2024-09-18T00:34:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2990 2024-09-18T00:34:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:34:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-18T00:34:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:34:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:34:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary7': 2024-09-18T00:34:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:34:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:34:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:34:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 39d491dc319d 2024-09-18T00:34:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.39d491dc319d 2024-09-18T00:34:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:34:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:34:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:34:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:34:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:34:36 2024 2024-09-18T00:34:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:34:36 2024 2024-09-18T00:34:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:34:36 2024 2024-09-18T00:34:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:34:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:34:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: f1fe4e73-72ac-4f20-85b7-52652fb1cf4b 2024-09-18T00:34:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:34:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:34:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:34:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary7 2024-09-18T00:34:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd6 2024-09-18T00:34:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:34:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd6 2024-09-18T00:34:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:34:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:34:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:34:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: ec2c1d49-2523-46c8-87ed-84f066dbb72d 2024-09-18T00:34:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:34:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:34:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:34:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:34:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:34:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:34:37.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary7 2024-09-18T00:34:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd6 test-primary7 2024-09-18T00:34:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:34:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary7 2024-09-18T00:34:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:34:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-18T00:34:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:34:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:34:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary7 2024-09-18T00:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-18T00:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary8 2024-09-18T00:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-18T00:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary7 2024-09-18T00:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-18T00:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-18T00:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:37.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:34:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:34:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:34:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:34:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-18T00:34:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:34:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:34:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-18T00:34:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:34:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-18T00:34:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:34:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-18T00:34:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:34:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:34:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2989 2024-09-18T00:34:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:34:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2993 2024-09-18T00:34:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-18T00:34:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary8': 2024-09-18T00:34:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:34:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:34:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:34:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 39e587627000 2024-09-18T00:34:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.39e587627000 2024-09-18T00:34:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:34:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:34:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:34:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:34:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:34:37 2024 2024-09-18T00:34:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:34:37 2024 2024-09-18T00:34:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:34:37 2024 2024-09-18T00:34:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:34:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:34:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 5dd62c2e-254c-4272-abb3-5adeb3b07fac 2024-09-18T00:34:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:34:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:34:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:34:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary8 2024-09-18T00:34:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd7 2024-09-18T00:34:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:34:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd7 2024-09-18T00:34:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:34:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:34:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:34:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: ee2a36c5-c49e-4ade-a97c-e95259412bc2 2024-09-18T00:34:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:34:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:34:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:34:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:34:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:34:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:34:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary8 2024-09-18T00:34:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd7 test-primary8 2024-09-18T00:34:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:34:38.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary8 2024-09-18T00:34:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:34:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-18T00:34:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:34:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:34:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary8 2024-09-18T00:34:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:34:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-18T00:34:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary9 2024-09-18T00:34:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-18T00:34:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:34:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:34:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:34:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary8 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:34:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-18T00:34:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:34:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:34:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-18T00:34:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:34:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-18T00:34:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:34:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-18T00:34:39.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2992 2024-09-18T00:34:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:34:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:34:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:34:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2996 2024-09-18T00:34:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-18T00:34:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary9': 2024-09-18T00:34:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:34:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:34:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:34:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 39f7c09f1c2c 2024-09-18T00:34:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.39f7c09f1c2c 2024-09-18T00:34:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:34:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:34:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:34:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:34:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:34:38 2024 2024-09-18T00:34:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:34:38 2024 2024-09-18T00:34:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:34:38 2024 2024-09-18T00:34:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:34:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:34:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: c9e2c879-a8a6-4001-8b94-cf4dbb212334 2024-09-18T00:34:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:34:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:34:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:34:39.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary9 2024-09-18T00:34:39.402 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:39.398+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-18T00:34:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd8 2024-09-18T00:34:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:34:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd8 2024-09-18T00:34:39.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:34:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:34:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:34:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: 1a06cc84-cf4d-4e26-b996-676b9db221a8 2024-09-18T00:34:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:34:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:34:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:34:39.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:34:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:34:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:34:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:39.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary9 2024-09-18T00:34:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd8 test-primary9 2024-09-18T00:34:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:34:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary9 2024-09-18T00:34:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:34:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-18T00:34:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:34:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:34:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:34:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary9 2024-09-18T00:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-18T00:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ MNTPT=test-primary10 2024-09-18T00:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-18T00:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary9 2024-09-18T00:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-18T00:34:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:34:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n snapshot ']' 2024-09-18T00:34:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-18T00:34:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:34:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local size=128 2024-09-18T00:34:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 10G ']' 2024-09-18T00:34:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ size=10G 2024-09-18T00:34:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ shift 2024-09-18T00:34:39.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-18T00:34:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ is_pool_mirror_mode_image mirror 2024-09-18T00:34:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' image = image ']' 2024-09-18T00:34:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:34:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-18T00:34:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:34:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mode=snapshot 2024-09-18T00:34:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-18T00:34:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:34:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-18T00:34:39.821 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:39.820+0000 7f9bfab49640 -1 reset not still connected to 0x55cd07747ba0 2024-09-18T00:34:39.823 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:39.820+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782eea0 2024-09-18T00:34:39.823 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:39.821+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-18T00:34:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2995 2024-09-18T00:34:39.956 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:39.952+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0ae1e820 2024-09-18T00:34:39.956 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:39.952+0000 7f9bfab49640 -1 reset not still connected to 0x55cd13d368f0 2024-09-18T00:34:39.956 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:39.954+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1ee4cdd0 2024-09-18T00:34:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:39.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2999 2024-09-18T00:34:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:34:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:34:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:34:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:34:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:34:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:34:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:41.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:34:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:34:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:34:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:34:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Mirroring enabled 2024-09-18T00:34:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-18T00:34:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:rbd image 'image-primary10': 2024-09-18T00:34:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: size 10 GiB in 2560 objects 2024-09-18T00:34:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: order 22 (4 MiB objects) 2024-09-18T00:34:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: snapshot_count: 1 2024-09-18T00:34:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: id: 3a096a89bc16 2024-09-18T00:34:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: block_name_prefix: rbd_data.3a096a89bc16 2024-09-18T00:34:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: format: 2 2024-09-18T00:34:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-18T00:34:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: op_features: 2024-09-18T00:34:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: flags: 2024-09-18T00:34:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: create_timestamp: Wed Sep 18 00:34:39 2024 2024-09-18T00:34:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: access_timestamp: Wed Sep 18 00:34:39 2024 2024-09-18T00:34:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: modify_timestamp: Wed Sep 18 00:34:39 2024 2024-09-18T00:34:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring state: enabled 2024-09-18T00:34:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring mode: snapshot 2024-09-18T00:34:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring global id: 1904d784-4580-4531-92b4-2a186a933d78 2024-09-18T00:34:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: mirroring primary: true 2024-09-18T00:34:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:34:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:34:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster1 device map -t krbd mirror/image-primary10 2024-09-18T00:34:41.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEV=/dev/rbd9 2024-09-18T00:34:41.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ DEVS+=($DEV) 2024-09-18T00:34:41.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mkfs.ext4 /dev/rbd9 2024-09-18T00:34:41.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-18T00:34:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Discarding device blocks: 0/2621440 done 2024-09-18T00:34:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-18T00:34:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Filesystem UUID: e7baf6e3-0ff4-4ee4-b125-564fba235279 2024-09-18T00:34:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Superblock backups stored on blocks: 2024-09-18T00:34:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-18T00:34:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Allocating group tables: 0/80 done 2024-09-18T00:34:41.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing inode tables: 0/80 done 2024-09-18T00:34:41.489 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Creating journal (16384 blocks): done 2024-09-18T00:34:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-18T00:34:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: 2024-09-18T00:34:41.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mkdir test-primary10 2024-09-18T00:34:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo mount /dev/rbd9 test-primary10 2024-09-18T00:34:41.652 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ whoami 2024-09-18T00:34:41.653 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo chown ubuntu test-primary10 2024-09-18T00:34:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ SNAP_PIDS+=($!) 2024-09-18T00:34:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-18T00:34:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-18T00:34:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:34:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:34:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ slow_untar_workload test-primary10 2024-09-18T00:34:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:34:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 142031 2024-09-18T00:34:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:41.665 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:34:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:34:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mountpt=test-primary10 2024-09-18T00:34:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-18T00:34:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local ret=0 2024-09-18T00:34:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-18T00:34:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 2998 2024-09-18T00:34:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3002 2024-09-18T00:34:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3001 2024-09-18T00:34:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:34:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:34:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3005 2024-09-18T00:34:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:34:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:34:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:34:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:34:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:34:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:34:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:34:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:34:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3003 2024-09-18T00:34:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3004 2024-09-18T00:34:43.337 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:44.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3007 2024-09-18T00:34:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3006 2024-09-18T00:34:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3008 2024-09-18T00:34:44.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:44.172 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:34:44.171+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb5790 2024-09-18T00:34:44.175 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:34:44.174+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ea270 2024-09-18T00:34:44.184 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:34:44.182+0000 7f706e974640 -1 reset not still connected to 0x55df1e1ead00 2024-09-18T00:34:44.184 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:34:44.182+0000 7f706e974640 -1 reset not still connected to 0x55df1e1eb450 2024-09-18T00:34:44.184 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:34:44.182+0000 7f706e974640 -1 reset not still connected to 0x55df3b9f09c0 2024-09-18T00:34:44.784 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:44.779+0000 7f9bfab49640 -1 reset not still connected to 0x55cd13d368f0 2024-09-18T00:34:44.784 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:44.781+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1ee4cdd0 2024-09-18T00:34:44.784 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:44.781+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3f313c70 2024-09-18T00:34:44.784 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:44.781+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3f5e28f0 2024-09-18T00:34:44.784 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:44.781+0000 7f9bfab49640 -1 reset not still connected to 0x55cd54e03040 2024-09-18T00:34:44.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3009 2024-09-18T00:34:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:45.132 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:34:45.130+0000 7f0114255640 -1 reset not still connected to 0x557d09ccdd40 2024-09-18T00:34:45.132 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:34:45.130+0000 7f0114255640 -1 reset not still connected to 0x557d09df6b60 2024-09-18T00:34:45.133 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:34:45.130+0000 7f0114255640 -1 reset not still connected to 0x557d09df6d00 2024-09-18T00:34:45.135 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:34:45.134+0000 7f0114255640 -1 reset not still connected to 0x557d0a065ee0 2024-09-18T00:34:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:34:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:34:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:34:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:34:45.145 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:34:45.143+0000 7f0114255640 -1 reset not still connected to 0x557d0a198d00 2024-09-18T00:34:45.145 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:34:45.143+0000 7f0114255640 -1 reset not still connected to 0x557d0b464750 2024-09-18T00:34:45.145 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:34:45.143+0000 7f0114255640 -1 reset not still connected to 0x557d0df51c70 2024-09-18T00:34:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:34:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:34:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:34:45.231 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:34:45.229+0000 7fd413089640 -1 reset not still connected to 0x5604b151a1a0 2024-09-18T00:34:45.231 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:34:45.229+0000 7fd413089640 -1 reset not still connected to 0x5604b6dac1a0 2024-09-18T00:34:45.231 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:34:45.229+0000 7fd413089640 -1 reset not still connected to 0x5604bb1176c0 2024-09-18T00:34:45.231 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:34:45.229+0000 7fd413089640 -1 reset not still connected to 0x5604bd5e3a00 2024-09-18T00:34:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:34:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:34:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:34:46.135 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:34:46.134+0000 7fd413089640 -1 reset not still connected to 0x5604bb1176c0 2024-09-18T00:34:46.136 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:34:46.134+0000 7fd413089640 -1 reset not still connected to 0x5604bd5e3a00 2024-09-18T00:34:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3011 2024-09-18T00:34:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:34:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:34:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:34:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:46.345 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:34:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:34:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3013 2024-09-18T00:34:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:34:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:34:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3012 2024-09-18T00:34:47.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3010 2024-09-18T00:34:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:34:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:34:47.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:34:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:34:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:34:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:47.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:34:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:34:47.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:34:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3014 2024-09-18T00:34:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3015 2024-09-18T00:34:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:34:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:34:49.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3016 2024-09-18T00:34:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3017 2024-09-18T00:34:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3018 2024-09-18T00:34:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3019 2024-09-18T00:34:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:34:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:50.216 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:34:50.217 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:34:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:34:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:34:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:34:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:34:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:50.223 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:34:50.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3020 2024-09-18T00:34:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:51.037 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:34:51.033+0000 7f6acfc5f640 -1 reset not still connected to 0x555c2baf9ad0 2024-09-18T00:34:51.037 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:34:51.033+0000 7f6acfc5f640 -1 reset not still connected to 0x555c3d37dad0 2024-09-18T00:34:51.144 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:34:51.142+0000 7fc275b74640 -1 reset not still connected to 0x559cbebff380 2024-09-18T00:34:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:34:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:34:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:34:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:34:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:34:51.805 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:51.803+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-18T00:34:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3023 2024-09-18T00:34:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3022 2024-09-18T00:34:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3021 2024-09-18T00:34:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:34:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:34:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:34:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:34:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:34:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:34:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:34:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:34:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:34:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:34:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3025 2024-09-18T00:34:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:53.352 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:53.349+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782eea0 2024-09-18T00:34:53.352 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:34:53.350+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-18T00:34:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3028 2024-09-18T00:34:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3026 2024-09-18T00:34:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3027 2024-09-18T00:34:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3024 2024-09-18T00:34:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:34:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:53.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:34:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:53.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3029 2024-09-18T00:34:54.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:34:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:34:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:34:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:34:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:34:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:34:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:34:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:34:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3030 2024-09-18T00:34:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:34:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:34:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:34:56.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3033 2024-09-18T00:34:56.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3032 2024-09-18T00:34:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3031 2024-09-18T00:34:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:34:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:34:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:34:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:34:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:34:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:34:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:34:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:34:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:34:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:34:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:57.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:34:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:34:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:34:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:34:57.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:34:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3038 2024-09-18T00:34:58.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3034 2024-09-18T00:34:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3035 2024-09-18T00:34:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3036 2024-09-18T00:34:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3037 2024-09-18T00:34:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:34:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:34:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:34:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:59.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:59.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3039 2024-09-18T00:34:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:34:59.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:34:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:34:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:34:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:34:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:34:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:35:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:35:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:35:00.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:35:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:35:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:35:00.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:35:00.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3040 2024-09-18T00:35:01.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:35:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:35:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:35:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3041 2024-09-18T00:35:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3043 2024-09-18T00:35:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3042 2024-09-18T00:35:02.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:02.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:35:02.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:02.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:02.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:35:02.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:35:02.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:02.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:35:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:35:02.012 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:35:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:35:02.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:35:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:35:02.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:35:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:35:02.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:35:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:35:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:35:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:35:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3047 2024-09-18T00:35:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3046 2024-09-18T00:35:03.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3044 2024-09-18T00:35:03.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3045 2024-09-18T00:35:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3048 2024-09-18T00:35:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:35:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:35:04.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:35:04.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3049 2024-09-18T00:35:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:35:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:35:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:35:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:35:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:35:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:35:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:35:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:35:05.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:35:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3050 2024-09-18T00:35:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:06.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:35:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:35:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:35:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3052 2024-09-18T00:35:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3051 2024-09-18T00:35:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3053 2024-09-18T00:35:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:35:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:35:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:35:07.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:07.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:35:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:35:07.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:35:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:35:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:35:07.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:35:07.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:35:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:35:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:35:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:07.947 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:35:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:35:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:35:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3054 2024-09-18T00:35:08.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:08.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3058 2024-09-18T00:35:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:08.195 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:35:08.193+0000 7fc275b74640 -1 reset not still connected to 0x559cbebc1110 2024-09-18T00:35:08.195 INFO:tasks.ceph.cluster2.osd.0.smithi192.stderr:2024-09-18T00:35:08.193+0000 7fc275b74640 -1 reset not still connected to 0x559cbebff380 2024-09-18T00:35:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3057 2024-09-18T00:35:08.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3055 2024-09-18T00:35:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3056 2024-09-18T00:35:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:35:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:35:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:35:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3059 2024-09-18T00:35:09.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:35:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:35:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:35:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:35:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:35:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:35:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:35:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:35:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:35:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3060 2024-09-18T00:35:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:35:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:35:11.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:35:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:35:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:35:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3061 2024-09-18T00:35:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3062 2024-09-18T00:35:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:35:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:35:12.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:35:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:35:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:35:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:35:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:35:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:35:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:35:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:35:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:35:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:35:13.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3063 2024-09-18T00:35:13.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3064 2024-09-18T00:35:13.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:13.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:35:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:35:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:35:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3065 2024-09-18T00:35:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3066 2024-09-18T00:35:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3067 2024-09-18T00:35:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3068 2024-09-18T00:35:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:14.067 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:35:14.064+0000 7f9bfab49640 -1 reset not still connected to 0x55cd092e1c70 2024-09-18T00:35:14.067 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:35:14.064+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0dc2cea0 2024-09-18T00:35:14.067 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:35:14.064+0000 7f9bfab49640 -1 reset not still connected to 0x55cd260cd6c0 2024-09-18T00:35:14.067 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:35:14.064+0000 7f9bfab49640 -1 reset not still connected to 0x55cd313d7e10 2024-09-18T00:35:14.069 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:35:14.066+0000 7f9bfab49640 -1 reset not still connected to 0x55cd4fbfd860 2024-09-18T00:35:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3069 2024-09-18T00:35:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:35:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:35:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:35:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:35:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:35:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:35:16.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3070 2024-09-18T00:35:16.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:35:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:35:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:35:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:35:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:35:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:35:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3072 2024-09-18T00:35:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3071 2024-09-18T00:35:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:35:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:35:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:35:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:35:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:35:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:35:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:17.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:35:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:35:17.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:35:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3073 2024-09-18T00:35:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:35:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:35:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:35:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3074 2024-09-18T00:35:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3078 2024-09-18T00:35:18.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3076 2024-09-18T00:35:19.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3075 2024-09-18T00:35:19.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3077 2024-09-18T00:35:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:35:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:35:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:35:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3079 2024-09-18T00:35:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:35:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:35:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:35:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:35:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:35:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:35:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:35:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3080 2024-09-18T00:35:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:35:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:35:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:35:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:35:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:35:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:35:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3082 2024-09-18T00:35:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3083 2024-09-18T00:35:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3081 2024-09-18T00:35:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:35:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:22.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:35:22.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:35:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:35:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:35:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:35:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:35:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:35:22.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:35:22.119 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:35:22.118+0000 7f0114255640 -1 reset not still connected to 0x557d0a198d00 2024-09-18T00:35:22.119 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:35:22.118+0000 7f0114255640 -1 reset not still connected to 0x557d0b464750 2024-09-18T00:35:22.120 INFO:tasks.ceph.cluster2.osd.2.smithi192.stderr:2024-09-18T00:35:22.118+0000 7f0114255640 -1 reset not still connected to 0x557d0df51c70 2024-09-18T00:35:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3084 2024-09-18T00:35:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3085 2024-09-18T00:35:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:35:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:35:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:35:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:23.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3088 2024-09-18T00:35:23.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3086 2024-09-18T00:35:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:35:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:23.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:35:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:35:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3087 2024-09-18T00:35:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:24.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3089 2024-09-18T00:35:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:35:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:35:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:35:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:35:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:35:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:35:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:35:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:35:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:35:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3090 2024-09-18T00:35:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:35:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:35:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:35:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:35:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:35:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:35:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3091 2024-09-18T00:35:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3092 2024-09-18T00:35:26.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3093 2024-09-18T00:35:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:35:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:35:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:35:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:35:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:35:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:35:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:35:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:35:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:35:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3095 2024-09-18T00:35:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3094 2024-09-18T00:35:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:35:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:35:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:35:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3098 2024-09-18T00:35:29.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:35:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:35:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:35:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3099 2024-09-18T00:35:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3096 2024-09-18T00:35:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3097 2024-09-18T00:35:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:35:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:35:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:35:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:35:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:35:29.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:35:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:35:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:30.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:35:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:35:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3100 2024-09-18T00:35:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:35:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:35:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:35:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:35:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:35:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:35:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3102 2024-09-18T00:35:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3101 2024-09-18T00:35:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3103 2024-09-18T00:35:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:35:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:35:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:35:32.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:32.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:35:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:35:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:35:32.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3105 2024-09-18T00:35:32.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3104 2024-09-18T00:35:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:35:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:35:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:35:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:35:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:35:32.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:35:33.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:33.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3106 2024-09-18T00:35:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:35:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:35:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:35:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3107 2024-09-18T00:35:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3109 2024-09-18T00:35:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3108 2024-09-18T00:35:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:35:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:35:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:35:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:35:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:35:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:35:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:35:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:35:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:35:35.995 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:35:35.993+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0dc2cea0 2024-09-18T00:35:35.995 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:35:35.993+0000 7f9bfab49640 -1 reset not still connected to 0x55cd260cd6c0 2024-09-18T00:35:35.995 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:35:35.993+0000 7f9bfab49640 -1 reset not still connected to 0x55cd313d7e10 2024-09-18T00:35:35.995 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:35:35.993+0000 7f9bfab49640 -1 reset not still connected to 0x55cd4fbfd860 2024-09-18T00:35:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3110 2024-09-18T00:35:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:35:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:35:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:35:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:35:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:35:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:35:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:36.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3111 2024-09-18T00:35:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3113 2024-09-18T00:35:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:35:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:35:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:35:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:35:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:35:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:35:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:35:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:35:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:35:37.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:35:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:35:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:35:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3114 2024-09-18T00:35:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3115 2024-09-18T00:35:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3112 2024-09-18T00:35:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:38.073 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:35:38.070+0000 7f6acfc5f640 -1 reset not still connected to 0x555c32872680 2024-09-18T00:35:38.073 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-18T00:35:38.070+0000 7f6acfc5f640 -1 reset not still connected to 0x555c40603790 2024-09-18T00:35:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3118 2024-09-18T00:35:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3116 2024-09-18T00:35:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3117 2024-09-18T00:35:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:35:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:35:39.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:35:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:39.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3119 2024-09-18T00:35:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:35:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:35:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:35:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:35:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:35:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:35:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3120 2024-09-18T00:35:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:35:41.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:35:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:35:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:35:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:35:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:35:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:35:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:35:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3121 2024-09-18T00:35:42.002 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3122 2024-09-18T00:35:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:35:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:35:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:35:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:35:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:35:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:35:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:35:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:35:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:35:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3125 2024-09-18T00:35:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:35:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:35:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:35:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3124 2024-09-18T00:35:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3123 2024-09-18T00:35:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3128 2024-09-18T00:35:44.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3126 2024-09-18T00:35:44.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3127 2024-09-18T00:35:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:35:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:35:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:35:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3129 2024-09-18T00:35:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:35:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:35:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:35:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:35:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:35:45.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:35:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:35:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:35:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:35:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3130 2024-09-18T00:35:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:35:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:35:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:35:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:35:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:35:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:35:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3131 2024-09-18T00:35:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3133 2024-09-18T00:35:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3132 2024-09-18T00:35:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:35:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:35:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:35:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:35:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:35:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:35:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:35:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:35:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:35:47.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:35:47.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:35:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:35:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3135 2024-09-18T00:35:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3134 2024-09-18T00:35:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:48.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:49.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3136 2024-09-18T00:35:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:35:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:35:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:35:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3138 2024-09-18T00:35:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3137 2024-09-18T00:35:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:49.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:49.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:49.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3139 2024-09-18T00:35:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:35:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:35:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:35:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:35:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:35:50.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:35:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:35:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:35:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:35:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:35:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:35:51.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:35:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:35:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:35:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:35:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3140 2024-09-18T00:35:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3141 2024-09-18T00:35:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:35:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:35:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:35:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3143 2024-09-18T00:35:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3142 2024-09-18T00:35:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:35:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:35:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:35:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:35:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:35:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:35:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:52.108 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:35:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:35:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:35:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3145 2024-09-18T00:35:53.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3144 2024-09-18T00:35:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:53.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3147 2024-09-18T00:35:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3146 2024-09-18T00:35:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3148 2024-09-18T00:35:54.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:54.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3149 2024-09-18T00:35:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:55.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:55.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:35:55.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:55.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:55.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:35:55.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:35:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:35:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:35:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:35:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:35:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:35:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:35:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:35:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:35:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:35:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3150 2024-09-18T00:35:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:35:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:35:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:35:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:35:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:35:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:35:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3152 2024-09-18T00:35:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3151 2024-09-18T00:35:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3153 2024-09-18T00:35:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:35:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:35:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:35:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:35:57.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:57.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:57.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:35:57.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:35:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:57.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:57.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:57.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:35:57.118 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:35:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:35:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:35:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:35:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:35:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3154 2024-09-18T00:35:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3155 2024-09-18T00:35:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3156 2024-09-18T00:35:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3158 2024-09-18T00:35:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3157 2024-09-18T00:35:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:35:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:35:59.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:35:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:35:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:35:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:35:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:59.111 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:35:59.935 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3159 2024-09-18T00:35:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:36:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:36:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:36:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:36:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:36:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:36:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:36:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:36:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:36:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3160 2024-09-18T00:36:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:36:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:36:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:36:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:36:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:36:01.112 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:36:01.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3163 2024-09-18T00:36:01.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3161 2024-09-18T00:36:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:36:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:36:02.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:36:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:02.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:36:02.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:02.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:02.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:36:02.114 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:36:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:36:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:36:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:36:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3165 2024-09-18T00:36:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3164 2024-09-18T00:36:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3162 2024-09-18T00:36:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:36:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:36:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:36:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3166 2024-09-18T00:36:03.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3168 2024-09-18T00:36:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3167 2024-09-18T00:36:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:36:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:36:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:36:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:04.939 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3169 2024-09-18T00:36:04.960 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:04.957+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3f5e28f0 2024-09-18T00:36:04.961 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:04.957+0000 7f9bfab49640 -1 reset not still connected to 0x55cd54e03040 2024-09-18T00:36:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:36:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:36:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:36:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:36:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:04.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:36:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:36:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3170 2024-09-18T00:36:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:36:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:36:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:36:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:36:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:36:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:36:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:36:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:36:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:36:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3171 2024-09-18T00:36:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3172 2024-09-18T00:36:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:36:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:36:06.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:36:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:36:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:06.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:36:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:36:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:06.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:36:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:36:06.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:36:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:06.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3175 2024-09-18T00:36:07.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:07.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3173 2024-09-18T00:36:07.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3174 2024-09-18T00:36:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:36:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:36:07.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:36:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:08.015 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:08.012+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1ee4cdd0 2024-09-18T00:36:08.015 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:08.012+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3f313c70 2024-09-18T00:36:08.015 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:08.012+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3f5e28f0 2024-09-18T00:36:08.015 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:08.012+0000 7f9bfab49640 -1 reset not still connected to 0x55cd54e03040 2024-09-18T00:36:08.015 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:08.012+0000 7f9bfab49640 -1 reset not still connected to 0x55cd07747ba0 2024-09-18T00:36:08.015 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:08.012+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782eea0 2024-09-18T00:36:08.015 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:08.012+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0782f1e0 2024-09-18T00:36:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3177 2024-09-18T00:36:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3176 2024-09-18T00:36:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:36:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:36:08.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:36:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3178 2024-09-18T00:36:09.943 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:36:09.941+0000 7f706e974640 -1 reset not still connected to 0x55df1dfb4340 2024-09-18T00:36:09.944 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:36:09.941+0000 7f706e974640 -1 reset not still connected to 0x55df288bf450 2024-09-18T00:36:09.946 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-18T00:36:09.944+0000 7f706e974640 -1 reset not still connected to 0x55df2cd6f1e0 2024-09-18T00:36:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:36:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:36:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:36:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:36:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:36:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:36:10.012 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:10.010+0000 7f9bfab49640 -1 reset not still connected to 0x55cd1ee4cdd0 2024-09-18T00:36:10.012 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:10.010+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3f313c70 2024-09-18T00:36:10.012 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:10.010+0000 7f9bfab49640 -1 reset not still connected to 0x55cd3f5e28f0 2024-09-18T00:36:10.013 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:10.010+0000 7f9bfab49640 -1 reset not still connected to 0x55cd54e03040 2024-09-18T00:36:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:36:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:36:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:36:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3180 2024-09-18T00:36:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3179 2024-09-18T00:36:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:36:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:36:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:36:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:36:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:36:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:36:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3183 2024-09-18T00:36:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3182 2024-09-18T00:36:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3181 2024-09-18T00:36:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:36:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:36:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:36:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:36:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:12.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:36:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:36:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:36:12.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3184 2024-09-18T00:36:12.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3185 2024-09-18T00:36:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:13.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3187 2024-09-18T00:36:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3186 2024-09-18T00:36:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:36:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:36:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:36:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:36:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:14.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:36:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:36:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3188 2024-09-18T00:36:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3189 2024-09-18T00:36:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:36:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:36:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:36:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:36:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:36:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:36:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:36:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:36:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:36:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3190 2024-09-18T00:36:16.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:16.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:36:16.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:16.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:16.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:36:16.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:36:16.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:36:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:36:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:36:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3192 2024-09-18T00:36:16.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3193 2024-09-18T00:36:16.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3191 2024-09-18T00:36:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:36:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:36:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:36:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:36:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:36:17.015 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:36:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:17.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:36:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:17.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:36:17.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:36:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:36:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:36:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:36:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3194 2024-09-18T00:36:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3195 2024-09-18T00:36:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3197 2024-09-18T00:36:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3198 2024-09-18T00:36:19.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3196 2024-09-18T00:36:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:36:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:36:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:36:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3199 2024-09-18T00:36:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:36:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:36:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:36:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:36:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:36:20.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:36:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:36:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:36:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:36:21.014 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3200 2024-09-18T00:36:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:36:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:36:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:36:21.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3202 2024-09-18T00:36:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3201 2024-09-18T00:36:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3203 2024-09-18T00:36:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:36:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:36:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:36:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:36:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:36:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:36:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:36:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:36:22.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:36:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:36:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:36:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:36:22.068 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:22.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:36:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:36:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:36:23.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3204 2024-09-18T00:36:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3206 2024-09-18T00:36:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3205 2024-09-18T00:36:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3207 2024-09-18T00:36:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3208 2024-09-18T00:36:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:36:24.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:36:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:36:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:24.164 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:24.160+0000 7f9bfab49640 -1 reset not still connected to 0x55cd0dc2cea0 2024-09-18T00:36:24.164 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:24.161+0000 7f9bfab49640 -1 reset not still connected to 0x55cd313d7e10 2024-09-18T00:36:24.164 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-18T00:36:24.161+0000 7f9bfab49640 -1 reset not still connected to 0x55cd4fbfd860 2024-09-18T00:36:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3209 2024-09-18T00:36:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:36:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:36:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:36:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:36:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:36:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:36:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:36:25.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:36:25.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:36:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3210 2024-09-18T00:36:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:36:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:36:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:36:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:36:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:36:26.115 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:36:27.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3213 2024-09-18T00:36:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3212 2024-09-18T00:36:27.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3211 2024-09-18T00:36:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:36:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:36:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:36:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:27.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:36:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:36:27.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:36:27.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:36:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:36:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:36:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:36:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:36:27.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:36:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3214 2024-09-18T00:36:28.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3215 2024-09-18T00:36:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3216 2024-09-18T00:36:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3218 2024-09-18T00:36:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3217 2024-09-18T00:36:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:36:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:36:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:36:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3219 2024-09-18T00:36:29.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:36:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:36:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:36:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:36:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:36:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:36:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:36:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:36:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:36:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3220 2024-09-18T00:36:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:36:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:36:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:36:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:36:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:36:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:36:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3221 2024-09-18T00:36:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3222 2024-09-18T00:36:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3223 2024-09-18T00:36:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:36:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:36:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:36:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:36:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:36:32.069 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:36:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:36:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:36:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:36:32.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:32.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:32.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:36:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:36:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:36:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3225 2024-09-18T00:36:33.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3224 2024-09-18T00:36:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:33.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3226 2024-09-18T00:36:34.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3228 2024-09-18T00:36:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3227 2024-09-18T00:36:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:36:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:36:34.067 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:36:34.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:34.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3229 2024-09-18T00:36:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:36:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:36:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:36:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:36:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:36:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:36:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:36:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:36:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:36:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3230 2024-09-18T00:36:35.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:36:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:36:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:36:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:36:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:36:36.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:36:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3231 2024-09-18T00:36:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3232 2024-09-18T00:36:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3233 2024-09-18T00:36:36.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:36:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:36:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:36:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:36:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:36:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:36:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:36:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:36:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:36:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3235 2024-09-18T00:36:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3234 2024-09-18T00:36:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:36:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:36:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:36:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:37.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3236 2024-09-18T00:36:38.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3237 2024-09-18T00:36:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3238 2024-09-18T00:36:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:36:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:36:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:36:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3239 2024-09-18T00:36:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:36:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:36:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:36:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:36:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:36:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:36:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:36:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:36:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:36:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3240 2024-09-18T00:36:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:36:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:36:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:36:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:36:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:36:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:36:41.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3243 2024-09-18T00:36:41.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3241 2024-09-18T00:36:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3242 2024-09-18T00:36:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:36:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:36:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:36:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:36:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:36:41.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:36:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:36:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:36:42.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:36:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:42.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3245 2024-09-18T00:36:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:36:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:36:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:36:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3244 2024-09-18T00:36:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3246 2024-09-18T00:36:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3247 2024-09-18T00:36:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3248 2024-09-18T00:36:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:36:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:36:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:36:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3249 2024-09-18T00:36:44.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:36:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:36:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:36:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:36:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:36:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:36:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:36:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:36:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:36:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3250 2024-09-18T00:36:46.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:36:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:36:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:36:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:36:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:46.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:36:46.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:36:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3251 2024-09-18T00:36:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3253 2024-09-18T00:36:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3252 2024-09-18T00:36:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:36:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:36:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:36:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:36:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:36:47.007 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:36:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:36:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:36:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:36:47.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:47.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:47.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:36:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:36:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:36:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3254 2024-09-18T00:36:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3255 2024-09-18T00:36:48.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3256 2024-09-18T00:36:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3258 2024-09-18T00:36:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3257 2024-09-18T00:36:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:36:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:36:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:36:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3259 2024-09-18T00:36:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:36:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:36:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:36:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:36:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:36:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:36:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:36:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:36:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:36:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:36:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:36:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:36:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:36:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:36:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:36:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3260 2024-09-18T00:36:52.013 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3261 2024-09-18T00:36:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3263 2024-09-18T00:36:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:36:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:36:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:36:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3262 2024-09-18T00:36:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:36:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:36:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:36:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:36:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:36:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:36:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:36:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:36:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3264 2024-09-18T00:36:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3265 2024-09-18T00:36:53.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3266 2024-09-18T00:36:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3268 2024-09-18T00:36:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3267 2024-09-18T00:36:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3269 2024-09-18T00:36:54.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:36:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:36:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:36:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:36:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:36:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:36:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-18T00:36:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:36:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-18T00:36:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:36:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:36:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:36:56.066 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3270 2024-09-18T00:36:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:36:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:36:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:36:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-18T00:36:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:36:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-18T00:36:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3272 2024-09-18T00:36:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3273 2024-09-18T00:36:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3271 2024-09-18T00:36:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-18T00:36:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:36:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-18T00:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:36:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:36:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-18T00:36:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:36:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-18T00:36:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:36:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:36:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:36:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3275 2024-09-18T00:36:58.133 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3274 2024-09-18T00:36:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3277 2024-09-18T00:36:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3276 2024-09-18T00:36:59.054 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3278 2024-09-18T00:36:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:36:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-18T00:36:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:36:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:36:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:36:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-18T00:36:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:36:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3279 2024-09-18T00:36:59.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:37:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:37:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:37:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:37:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:37:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:37:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:37:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:37:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-18T00:37:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:37:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:37:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:37:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-18T00:37:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:37:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 142163 2024-09-18T00:37:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3280 2024-09-18T00:37:00.976 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:37:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:37:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-18T00:37:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:37:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:37:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:37:01.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-18T00:37:01.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:37:01.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 142295 2024-09-18T00:37:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3281 2024-09-18T00:37:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3282 2024-09-18T00:37:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:37:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:37:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:37:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 142427 2024-09-18T00:37:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:37:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 142579 2024-09-18T00:37:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:37:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-18T00:37:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:37:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:37:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:37:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-18T00:37:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3283 2024-09-18T00:37:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:37:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-18T00:37:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:37:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:37:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:37:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-18T00:37:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:37:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3284 2024-09-18T00:37:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:37:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 142731 2024-09-18T00:37:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:37:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3285 2024-09-18T00:37:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:37:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for i in {1..30} 2024-09-18T00:37:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-18T00:37:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:37:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:37:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:37:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 142884 2024-09-18T00:37:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:37:04.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-18T00:37:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:37:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 143037 2024-09-18T00:37:06.953 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Snapshot ID: 3286 2024-09-18T00:37:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 3 2024-09-18T00:37:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:37:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 143190 2024-09-18T00:37:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-18T00:37:07.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 143403 2024-09-18T00:37:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:37:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:37:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 142032 2024-09-18T00:39:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:39:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:39:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:39:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 142164 2024-09-18T00:39:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:39:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:39:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:39:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 142296 2024-09-18T00:39:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:39:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:39:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:39:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 142428 2024-09-18T00:39:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:39:34.582 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:39:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:39:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 142580 2024-09-18T00:39:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:39:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:39:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:39:35.457 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 142732 2024-09-18T00:39:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:39:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:39:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:39:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 142885 2024-09-18T00:39:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:39:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:39:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:39:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 143038 2024-09-18T00:39:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:39:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:39:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:39:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 143191 2024-09-18T00:39:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:39:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:39:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-18T00:39:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 143404 2024-09-18T00:39:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ret=124 2024-09-18T00:39:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( ret != 124 )) 2024-09-18T00:39:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:39:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:39:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:39:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:39:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 1 2024-09-18T00:39:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd0 2024-09-18T00:39:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary1 2024-09-18T00:39:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:39:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary1 2024-09-18T00:39:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:39:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary1 2024-09-18T00:39:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:39:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 2 2024-09-18T00:39:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd1 2024-09-18T00:39:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary2 2024-09-18T00:39:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary2 2024-09-18T00:39:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:39:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary2 2024-09-18T00:39:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:39:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 3 2024-09-18T00:39:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:39:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd2 2024-09-18T00:39:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary3 2024-09-18T00:39:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary3 2024-09-18T00:39:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:39:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary3 2024-09-18T00:39:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:39:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:39:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 4 2024-09-18T00:39:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd3 2024-09-18T00:39:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:39:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary4 2024-09-18T00:39:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary4 2024-09-18T00:39:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:39:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary4 2024-09-18T00:39:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:39:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 5 2024-09-18T00:39:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd4 2024-09-18T00:39:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:39:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary5 2024-09-18T00:39:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary5 2024-09-18T00:39:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:39:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:39:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary5 2024-09-18T00:39:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 6 2024-09-18T00:39:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd5 2024-09-18T00:39:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:39:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:39:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary6 2024-09-18T00:39:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary6 2024-09-18T00:39:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:39:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary6 2024-09-18T00:39:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 7 2024-09-18T00:39:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd6 2024-09-18T00:39:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary7 2024-09-18T00:39:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:39:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary7 2024-09-18T00:39:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:39:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary7 2024-09-18T00:39:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:39:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:39:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:39:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ COMPARE_PIDS+=($!) 2024-09-18T00:39:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:39:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 150229 2024-09-18T00:39:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 10 2024-09-18T00:39:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd9 2024-09-18T00:39:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary10 2024-09-18T00:39:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary10 2024-09-18T00:39:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:39:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary10 2024-09-18T00:39:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 9 2024-09-18T00:39:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd8 2024-09-18T00:39:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary9 2024-09-18T00:39:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary9 2024-09-18T00:39:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:39:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary9 2024-09-18T00:39:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ compare_demoted_promoted_image 8 2024-09-18T00:39:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local dev=/dev/rbd7 2024-09-18T00:39:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local img=image-primary8 2024-09-18T00:39:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local mntpt=test-primary8 2024-09-18T00:39:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local demote_md5 promote_md5 2024-09-18T00:39:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo umount test-primary8 2024-09-18T00:39:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-18T00:39:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:39:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-18T00:39:42.178 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:39:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:39:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-18T00:39:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-18T00:39:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:39:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-18T00:39:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:39:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-18T00:39:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:39:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:39:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-18T00:39:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-18T00:39:42.338 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:39:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-18T00:39:42.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:39:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-18T00:39:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:40:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=80e98a44f77448a9bf8a37afd80098ac 2024-09-18T00:40:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary7 2024-09-18T00:40:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-18T00:40:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:40:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-18T00:40:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=82dd0c8592de0c89c93acf1980e8990b 2024-09-18T00:40:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary8 2024-09-18T00:40:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-18T00:40:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:40:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-18T00:40:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=612c8739f5e494e771b78c12886fff24 2024-09-18T00:40:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary3 2024-09-18T00:40:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:40:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-18T00:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-18T00:40:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:40:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-18T00:40:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=743b01b4b566e1fad0ef0d2b43de8ad3 2024-09-18T00:40:12.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary5 2024-09-18T00:40:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-18T00:40:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:40:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-18T00:40:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=ab23ae8f998c7f9f1a8d66ae6f5b91d8 2024-09-18T00:40:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary2 2024-09-18T00:40:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-18T00:40:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:40:13.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-18T00:40:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-18T00:40:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:40:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:40:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=bd8f94c97af40cd24efa56df5dd14767 2024-09-18T00:40:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary10 2024-09-18T00:40:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-18T00:40:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:40:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-18T00:40:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:40:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: f1fe4e73-72ac-4f20-85b7-52652fb1cf4b 2024-09-18T00:40:13.620 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:12 2024-09-18T00:40:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:40:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:13' 2024-09-18T00:40:13.621 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:13.623 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:13.624 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:12' 2024-09-18T00:40:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-18 00:40:12' 2024-09-18T00:40:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:40:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:12' 2024-09-18T00:40:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:13.637 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=81b8da86565042b63c55ec59015593c8 2024-09-18T00:40:13.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary1 2024-09-18T00:40:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=5b8e6217970ed1a21f0dfe25c723901a 2024-09-18T00:40:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary4 2024-09-18T00:40:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-18T00:40:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:40:13.734 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-18T00:40:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1942634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:40:12cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3018752.0,"last_snapshot_bytes":5898240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619816,"remote_snapshot_timestamp":1726619816,"replay_state":"idle"}2024-09-18 00:39:47image-primary101904d784-4580-4531-92b4-2a186a933d78up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:40:12cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3665920.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619825,"remote_snapshot_timestamp":1726619825,"replay_state":"idle"}2024-09-18 00:39:47image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:40:12cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":126976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619817,"remote_snapshot_timestamp":1726619817,"replay_state":"idle"}2024-09-18 00:39:47image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:40:12cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4157440.0,"last_snapshot_bytes":8155136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619818,"remote_snapshot_timestamp":1726619818,"replay_state":"idle"}2024-09-18 00:39:47image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:40:12cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619818,"remote_snapshot_timestamp":1726619818,"replay_state":"idle"}2024-09-18 00:39:47image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:40:12cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3352576.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619819,"remote_snapshot_timestamp":1726619819,"replay_state":"idle"}2024-09-18 00:39:47image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:40:12cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":188416.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619820,"remote_snapshot_timestamp":1726619820,"replay_state":"idle"}2024-09-18 00:39:47image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:12cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image demoted2024-09-18 00:40:13image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:40:12cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3897344.0,"last_snapshot_bytes":6041600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619822,"remote_snapshot_timestamp":1726619822,"replay_state":"idle"}2024-09-18 00:39:47image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+stoppedlocal image is primary42634266mirror.0smithi1922024-09-18 00:40:12cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3368960.0,"last_snapshot_bytes":6598656,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619823,"remote_snapshot_timestamp":1726619823,"replay_state":"idle"}2024-09-18 00:39:47' 2024-09-18T00:40:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:40:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:12' 2024-09-18T00:40:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:40:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-18T00:40:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:40:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=dedaccc682d3d4b7ad75fc9a1da20541 2024-09-18T00:40:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary9 2024-09-18T00:40:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-18T00:40:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:40:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-18T00:40:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-18T00:40:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:40:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-18T00:40:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_md5=515ace2c9e31b5fcb65f30aec15b50e2 2024-09-18T00:40:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster1 device unmap -t krbd mirror/image-primary6 2024-09-18T00:40:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-18T00:40:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:40:14.018 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-18T00:40:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:40:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:40:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:40:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:40:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-18T00:40:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:40:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:14.584 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-18T00:40:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:40:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-18T00:40:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:40:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:14.614 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-18T00:40:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:14.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:40:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:14.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-18T00:40:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:40:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-18T00:40:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:40:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: f1fe4e73-72ac-4f20-85b7-52652fb1cf4b 2024-09-18T00:40:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:40:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:13 2024-09-18T00:40:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:40:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:12' 2024-09-18T00:40:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image demoted 2024-09-18T00:40:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:13' 2024-09-18T00:40:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-18 00:40:13' 2024-09-18T00:40:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:40:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:13' 2024-09-18T00:40:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:14.838 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING1942604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3018752.0,"last_snapshot_bytes":5898240,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619816,"remote_snapshot_timestamp":1726619816,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:39:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:40:12image-primary101904d784-4580-4531-92b4-2a186a933d78up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3665920.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619825,"remote_snapshot_timestamp":1726619825,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:39:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:40:12image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":126976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619817,"remote_snapshot_timestamp":1726619817,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:39:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:40:12image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4157440.0,"last_snapshot_bytes":8155136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619818,"remote_snapshot_timestamp":1726619818,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:39:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:40:12image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":139264.0,"last_snapshot_bytes":139264,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619818,"remote_snapshot_timestamp":1726619818,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:39:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:40:12image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3352576.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619819,"remote_snapshot_timestamp":1726619819,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:39:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:40:12image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":188416.0,"last_snapshot_bytes":143360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619820,"remote_snapshot_timestamp":1726619820,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:39:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:40:12image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+unknownremote image demoted42604266mirror.0smithi1922024-09-18 00:40:13cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:12image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3897344.0,"last_snapshot_bytes":6041600,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619822,"remote_snapshot_timestamp":1726619822,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:39:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:40:12image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3368960.0,"last_snapshot_bytes":6598656,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726619823,"remote_snapshot_timestamp":1726619823,"replay_state":"idle"}42604266mirror.0smithi1922024-09-18 00:39:47cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+stoppedlocal image is primary2024-09-18 00:40:12' 2024-09-18T00:40:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:40:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:13' 2024-09-18T00:40:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:40:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-18T00:40:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:40:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:40:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-18T00:40:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:40:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:40:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:40:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary7 2024-09-18T00:40:15.571 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:40:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:40:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:40:15.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:40:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-18T00:40:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:15.586 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:40:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:40:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image demoted to non-primary 2024-09-18T00:40:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-18T00:40:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:15.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:40:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:15.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:40:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-18T00:40:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:40:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:15.605 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-18T00:40:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:40:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:15.617 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-18T00:40:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-18T00:40:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:40:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:40:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-18T00:40:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:40:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-18T00:40:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-18T00:40:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:40:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:40:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-18T00:40:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:40:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:15.689 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:40:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9d6ca525-e53f-4cf8-80fc-35c7b5df9fac 2024-09-18T00:40:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:40:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:40:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:12 2024-09-18T00:40:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:40:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:15' 2024-09-18T00:40:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:40:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 5dd62c2e-254c-4272-abb3-5adeb3b07fac 2024-09-18T00:40:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:40:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:40:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:12 2024-09-18T00:40:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:40:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:15' 2024-09-18T00:40:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:40:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:40:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:40:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:40:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:40:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 91db1fb0-ad2a-47fe-bde5-20bf3e90ea37 2024-09-18T00:40:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:40:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:40:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:12 2024-09-18T00:40:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:40:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:15' 2024-09-18T00:40:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:40:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:40:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:40:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 6edb0522-635d-46cd-8cf6-76f54f3926a0 2024-09-18T00:40:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:40:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:40:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:12 2024-09-18T00:40:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:40:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:15' 2024-09-18T00:40:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:40:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:40:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd0 2024-09-18T00:40:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd0 2024-09-18T00:40:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:40:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-18T00:40:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:40:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:40:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:40:16.658 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c19919df-d5b3-43ec-abfb-adf02bda5de0 2024-09-18T00:40:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:40:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:40:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:12 2024-09-18T00:40:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:40:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:16' 2024-09-18T00:40:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:40:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:40:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-18T00:40:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:40:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:40:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-18T00:40:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:40:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-18T00:40:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-18T00:40:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:40:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:40:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-18T00:40:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:40:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:40:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:40:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 61594334-abd8-44fc-9715-2484fbb2143a 2024-09-18T00:40:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:40:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:40:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:12 2024-09-18T00:40:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:40:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:16' 2024-09-18T00:40:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:40:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:40:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:40:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a0539cb4-609b-47fd-8ae0-93b0a3b749f0 2024-09-18T00:40:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:40:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:40:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:12 2024-09-18T00:40:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:40:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:16' 2024-09-18T00:40:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:40:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:16.726 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:40:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:40:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1904d784-4580-4531-92b4-2a186a933d78 2024-09-18T00:40:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:40:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:40:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:12 2024-09-18T00:40:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:40:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:16' 2024-09-18T00:40:16.736 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:40:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:40:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:40:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c9e2c879-a8a6-4001-8b94-cf4dbb212334 2024-09-18T00:40:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:40:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:40:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:12 2024-09-18T00:40:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image demoted 2024-09-18T00:40:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:16' 2024-09-18T00:40:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 1 2024-09-18T00:40:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 2 2024-09-18T00:40:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-18T00:40:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:40:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:17.846 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:40:17.860 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-18T00:40:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:40:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:17.861 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:40:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-18T00:40:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:40:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:17.868 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:40:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:40:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 5dd62c2e-254c-4272-abb3-5adeb3b07fac 2024-09-18T00:40:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:17.908 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:40:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:17.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:40:17.919 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9d6ca525-e53f-4cf8-80fc-35c7b5df9fac 2024-09-18T00:40:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:40:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 91db1fb0-ad2a-47fe-bde5-20bf3e90ea37 2024-09-18T00:40:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-18T00:40:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:40:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:40:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-18T00:40:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:40:17.936 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:17.937 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:40:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 6edb0522-635d-46cd-8cf6-76f54f3926a0 2024-09-18T00:40:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:17.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:17.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:17.996 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:40:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:18.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:40:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:40:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:18.037 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-18T00:40:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:40:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:40:18.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:40:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:18.042 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:18.043 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.044 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-18T00:40:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:40:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:18.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:18.085 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:40:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:40:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-18T00:40:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:40:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:40:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:40:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-18T00:40:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:40:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:18.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-18T00:40:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:40:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:40:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:40:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c19919df-d5b3-43ec-abfb-adf02bda5de0 2024-09-18T00:40:18.711 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:18.712 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:18.715 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:18.716 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-18T00:40:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:40:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:40:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:40:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-18T00:40:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:40:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-18T00:40:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-18T00:40:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:40:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:40:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-18T00:40:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:40:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.749 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:40:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:40:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 61594334-abd8-44fc-9715-2484fbb2143a 2024-09-18T00:40:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:18.778 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:18.779 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:40:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:40:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a0539cb4-609b-47fd-8ae0-93b0a3b749f0 2024-09-18T00:40:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:40:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1904d784-4580-4531-92b4-2a186a933d78 2024-09-18T00:40:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:40:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:40:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:40:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:40:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:40:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c9e2c879-a8a6-4001-8b94-cf4dbb212334 2024-09-18T00:40:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-18T00:40:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:40:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:18.832 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:40:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:40:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:18.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:40:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:18.885 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:18.886 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:18.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-18T00:40:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:40:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:18.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:40:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:40:18.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:18.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:18.983 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:40:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:40:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-18T00:40:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:40:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:18.987 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:18.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-18T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42634266mirror.0smithi1922024-09-18 00:40:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:40:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:40:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-18T00:40:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:40:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-18T00:40:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:40:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-18T00:40:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-18T00:40:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:40:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-18T00:40:19.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-18T00:40:19.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.091 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:40:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:19.092 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-18T00:40:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:40:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 5dd62c2e-254c-4272-abb3-5adeb3b07fac 2024-09-18T00:40:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:40:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:40:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:19.101 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:40:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9d6ca525-e53f-4cf8-80fc-35c7b5df9fac 2024-09-18T00:40:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:19.102 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:40:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:19.104 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:40:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:40:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 6edb0522-635d-46cd-8cf6-76f54f3926a0 2024-09-18T00:40:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:40:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:19.155 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:19.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:19.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:19.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:40:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:19.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:19.169 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:40:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:19.235 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:40:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:19.240 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-18T00:40:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:40:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:40:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-18T00:40:19.250 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:40:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:40:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:19.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:19.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:19.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:19.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-18T00:40:19.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:40:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:40:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-18T00:40:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:40:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:40:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-18T00:40:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:40:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:40:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-18T00:40:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-18T00:40:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:40:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-18T00:40:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:40:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 91db1fb0-ad2a-47fe-bde5-20bf3e90ea37 2024-09-18T00:40:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:19.490 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:40:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:19.491 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:40:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:40:19.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:19.591 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:40:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-18T00:40:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:40:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:40:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-18T00:40:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-18T00:40:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:40:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-18T00:40:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:40:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:40:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:40:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:40:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:40:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:40:19.880 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary2 2024-09-18T00:40:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:40:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:40:19.881 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary5 2024-09-18T00:40:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:40:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:40:19.883 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary3 2024-09-18T00:40:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:40:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:40:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary8 2024-09-18T00:40:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-18T00:40:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:40:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-18T00:40:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:40:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c19919df-d5b3-43ec-abfb-adf02bda5de0 2024-09-18T00:40:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:40:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:40:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:19.921 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:40:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 61594334-abd8-44fc-9715-2484fbb2143a 2024-09-18T00:40:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:40:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:19.963 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:19.964 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:40:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-18T00:40:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.988 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:40:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:19.989 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-18T00:40:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-18T00:40:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:40:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-18T00:40:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-18T00:40:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:40:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-18T00:40:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd1 2024-09-18T00:40:20.030 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd1 2024-09-18T00:40:20.032 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:40:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd4 2024-09-18T00:40:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd4 2024-09-18T00:40:20.036 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:40:20.051 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd2 2024-09-18T00:40:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd2 2024-09-18T00:40:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:40:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd3 2024-09-18T00:40:20.081 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd3 2024-09-18T00:40:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:40:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:40:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a0539cb4-609b-47fd-8ae0-93b0a3b749f0 2024-09-18T00:40:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:20.163 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:40:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:20.164 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:20.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:20.172 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:20.174 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:20.176 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:20.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:20.184 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:20.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:40:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:40:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:20.192 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:20.193 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:40:20.195 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-18T00:40:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:40:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:40:20.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-18T00:40:20.208 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c9e2c879-a8a6-4001-8b94-cf4dbb212334 2024-09-18T00:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:20.222 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:20.231 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:40:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1904d784-4580-4531-92b4-2a186a933d78 2024-09-18T00:40:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17 2024-09-18T00:40:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster1 2024-09-18T00:40:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+unknown 2024-09-18T00:40:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: remote image is not primary 2024-09-18T00:40:20.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:17' 2024-09-18T00:40:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+unknown' 2024-09-18T00:40:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:20.262 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+unknown 2024-09-18T00:40:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:40:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:20.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:20.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: remote image is not primary 2024-09-18T00:40:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:20.270 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:20.271 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-18 00:40:17' 2024-09-18T00:40:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:40:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+unknown 2024-09-18T00:40:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:17' 2024-09-18T00:40:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:20.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:40:20.304 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:20.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:40:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:20.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-18T00:40:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:40:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:40:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-18T00:40:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:40:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:40:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:20.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:20.454 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-18T00:40:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:40:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:40:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-18T00:40:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:20.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:40:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:40:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-18T00:40:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:40:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:40:20.552 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-18T00:40:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='WARNINGOKWARNING9142604266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary101904d784-4580-4531-92b4-2a186a933d78up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+stoppedlocal image is primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}2024-09-18 00:40:17image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+unknownremote image is not primary42604266mirror.0smithi1922024-09-18 00:40:17cluster1ce7174d7-6418-49e8-9ec0-0a04ee4d9b77up+unknownremote image is not primary2024-09-18 00:40:17' 2024-09-18T00:40:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:40:20.569 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:17' 2024-09-18T00:40:20.570 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:40:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+unknown 2024-09-18T00:40:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+unknown 2024-09-18T00:40:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+unknown 2024-09-18T00:40:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+unknown 2024-09-18T00:40:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-18T00:40:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:40:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local force= 2024-09-18T00:40:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-18T00:40:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:40:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:40:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:40:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:40:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:Image promoted to primary 2024-09-18T00:40:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:40:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:40:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary9 2024-09-18T00:40:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:40:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:40:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:40:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:40:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary10 2024-09-18T00:40:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary1 2024-09-18T00:40:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:40:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:40:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \n\b\d ]] 2024-09-18T00:40:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ krbd == \k\r\b\d ]] 2024-09-18T00:40:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary4 2024-09-18T00:40:21.205 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo rbd --cluster cluster2 device map -t krbd mirror/image-primary6 2024-09-18T00:40:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd6 2024-09-18T00:40:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd5 2024-09-18T00:40:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd6 2024-09-18T00:40:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:40:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd5 2024-09-18T00:40:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:40:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd7 2024-09-18T00:40:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:40:21.441 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd7 2024-09-18T00:40:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd8 2024-09-18T00:40:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:40:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd8 2024-09-18T00:40:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ dev=/dev/rbd9 2024-09-18T00:40:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sudo md5sum /dev/rbd9 2024-09-18T00:40:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ awk '{print $1}' 2024-09-18T00:40:22.114 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:40:22.113+0000 7fd413089640 -1 reset not still connected to 0x5604abef9ba0 2024-09-18T00:40:22.115 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:40:22.113+0000 7fd413089640 -1 reset not still connected to 0x5604b02431e0 2024-09-18T00:40:22.115 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:40:22.113+0000 7fd413089640 -1 reset not still connected to 0x5604b7d9a8f0 2024-09-18T00:40:22.115 INFO:tasks.ceph.cluster2.osd.1.smithi192.stderr:2024-09-18T00:40:22.113+0000 7fd413089640 -1 reset not still connected to 0x5604d5b3aea0 2024-09-18T00:40:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=80e98a44f77448a9bf8a37afd80098ac 2024-09-18T00:40:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd0 2024-09-18T00:40:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 80e98a44f77448a9bf8a37afd80098ac != \8\0\e\9\8\a\4\4\f\7\7\4\4\8\a\9\b\f\8\a\3\7\a\f\d\8\0\0\9\8\a\c ]] 2024-09-18T00:40:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=82dd0c8592de0c89c93acf1980e8990b 2024-09-18T00:40:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd4 2024-09-18T00:40:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 82dd0c8592de0c89c93acf1980e8990b != \8\2\d\d\0\c\8\5\9\2\d\e\0\c\8\9\c\9\3\a\c\f\1\9\8\0\e\8\9\9\0\b ]] 2024-09-18T00:40:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=ab23ae8f998c7f9f1a8d66ae6f5b91d8 2024-09-18T00:40:51.139 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd2 2024-09-18T00:40:51.266 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ ab23ae8f998c7f9f1a8d66ae6f5b91d8 != \a\b\2\3\a\e\8\f\9\9\8\c\7\f\9\f\1\a\8\d\6\6\a\e\6\f\5\b\9\1\d\8 ]] 2024-09-18T00:40:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=515ace2c9e31b5fcb65f30aec15b50e2 2024-09-18T00:40:51.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd9 2024-09-18T00:40:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 515ace2c9e31b5fcb65f30aec15b50e2 != \5\1\5\a\c\e\2\c\9\e\3\1\b\5\f\c\b\6\5\f\3\0\a\e\c\1\5\b\5\0\e\2 ]] 2024-09-18T00:40:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=612c8739f5e494e771b78c12886fff24 2024-09-18T00:40:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd1 2024-09-18T00:40:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 612c8739f5e494e771b78c12886fff24 != \6\1\2\c\8\7\3\9\f\5\e\4\9\4\e\7\7\1\b\7\8\c\1\2\8\8\6\f\f\f\2\4 ]] 2024-09-18T00:40:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=dedaccc682d3d4b7ad75fc9a1da20541 2024-09-18T00:40:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd7 2024-09-18T00:40:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ dedaccc682d3d4b7ad75fc9a1da20541 != \d\e\d\a\c\c\c\6\8\2\d\3\d\4\b\7\a\d\7\5\f\c\9\a\1\d\a\2\0\5\4\1 ]] 2024-09-18T00:40:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=bd8f94c97af40cd24efa56df5dd14767 2024-09-18T00:40:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd5 2024-09-18T00:40:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=743b01b4b566e1fad0ef0d2b43de8ad3 2024-09-18T00:40:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd3 2024-09-18T00:40:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ bd8f94c97af40cd24efa56df5dd14767 != \b\d\8\f\9\4\c\9\7\a\f\4\0\c\d\2\4\e\f\a\5\6\d\f\5\d\d\1\4\7\6\7 ]] 2024-09-18T00:40:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 743b01b4b566e1fad0ef0d2b43de8ad3 != \7\4\3\b\0\1\b\4\b\5\6\6\e\1\f\a\d\0\e\f\0\d\2\b\4\3\d\e\8\a\d\3 ]] 2024-09-18T00:40:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=81b8da86565042b63c55ec59015593c8 2024-09-18T00:40:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd6 2024-09-18T00:40:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 81b8da86565042b63c55ec59015593c8 != \8\1\b\8\d\a\8\6\5\6\5\0\4\2\b\6\3\c\5\5\e\c\5\9\0\1\5\5\9\3\c\8 ]] 2024-09-18T00:40:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:40:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 150230 2024-09-18T00:40:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:40:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 150232 2024-09-18T00:40:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:40:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 150234 2024-09-18T00:40:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ promote_md5=5b8e6217970ed1a21f0dfe25c723901a 2024-09-18T00:40:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sudo rbd --cluster cluster2 device unmap -t krbd /dev/rbd8 2024-09-18T00:40:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ [[ 5b8e6217970ed1a21f0dfe25c723901a != \5\b\8\e\6\2\1\7\9\7\0\e\d\1\a\2\1\f\0\d\f\e\2\5\c\7\2\3\9\0\1\a ]] 2024-09-18T00:40:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:40:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 150236 2024-09-18T00:40:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:40:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 150238 2024-09-18T00:40:53.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 150240 2024-09-18T00:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 150242 2024-09-18T00:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 150244 2024-09-18T00:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-18T00:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait 150245 2024-09-18T00:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ (( RET != 0 )) 2024-09-18T00:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary1 2024-09-18T00:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary1 2024-09-18T00:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-18T00:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-18T00:40:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:40:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:40:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-18T00:40:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary1: 2024-09-18T00:40:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 61594334-abd8-44fc-9715-2484fbb2143a 2024-09-18T00:40:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:40:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3195904.0,"bytes_per_snapshot":63918080.0,"last_snapshot_bytes":63918080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"} 2024-09-18T00:40:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:47 2024-09-18T00:40:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:54.533 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:40:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:40:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:52' 2024-09-18T00:40:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:40:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:40:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3195904.0,"bytes_per_snapshot":63918080.0,"last_snapshot_bytes":63918080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"} 2024-09-18T00:40:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:54.535 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:40:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:40:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:40:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-18 00:40:47' 2024-09-18T00:40:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:40:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:40:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:47' 2024-09-18T00:40:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK1042634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary161594334-abd8-44fc-9715-2484fbb2143aup+replayingreplaying, {"bytes_per_second":3195904.0,"bytes_per_snapshot":63918080.0,"last_snapshot_bytes":63918080,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:52image-primary101904d784-4580-4531-92b4-2a186a933d78up+replayingreplaying, {"bytes_per_second":3243622.4,"bytes_per_snapshot":64872448.0,"last_snapshot_bytes":64872448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:52image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+replayingreplaying, {"bytes_per_second":3231334.4,"bytes_per_snapshot":64626688.0,"last_snapshot_bytes":64626688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:52image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+replayingreplaying, {"bytes_per_second":3162316.8,"bytes_per_snapshot":63246336.0,"last_snapshot_bytes":63246336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:52image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+replayingreplaying, {"bytes_per_second":3237683.2,"bytes_per_snapshot":64753664.0,"last_snapshot_bytes":64753664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:52image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+replayingreplaying, {"bytes_per_second":3181158.4,"bytes_per_snapshot":63623168.0,"last_snapshot_bytes":63623168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:52image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+replayingreplaying, {"bytes_per_second":3220889.6,"bytes_per_snapshot":64417792.0,"last_snapshot_bytes":64417792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:52image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":3354214.4,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620015,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:52image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+replayingreplaying, {"bytes_per_second":3345817.6,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:52image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+replayingreplaying, {"bytes_per_second":3074252.8,"bytes_per_snapshot":61485056.0,"last_snapshot_bytes":61485056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:52' 2024-09-18T00:40:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-18T00:40:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:40:54.629 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-18T00:40:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:40:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:40:54.631 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:40:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:40:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-18T00:40:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:40:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-18T00:40:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-18T00:40:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:40:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-18T00:40:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary1 2024-09-18T00:40:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:40:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary1 2024-09-18T00:40:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary1 2024-09-18T00:40:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:40:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary2 2024-09-18T00:40:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary2 2024-09-18T00:40:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:40:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:40:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-18T00:40:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:40:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:40:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:40:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-18T00:40:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:40:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:40:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-18T00:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary2: 2024-09-18T00:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 9d6ca525-e53f-4cf8-80fc-35c7b5df9fac 2024-09-18T00:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3231334.4,"bytes_per_snapshot":64626688.0,"last_snapshot_bytes":64626688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"} 2024-09-18T00:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:47 2024-09-18T00:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:40:57.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:52' 2024-09-18T00:40:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:40:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:40:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:40:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3231334.4,"bytes_per_snapshot":64626688.0,"last_snapshot_bytes":64626688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"} 2024-09-18T00:40:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:40:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:40:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:40:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:40:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:40:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:40:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:40:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:40:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:40:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-18 00:40:47' 2024-09-18T00:40:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:40:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:40:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:47' 2024-09-18T00:40:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:40:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:40:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:40:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK942634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary101904d784-4580-4531-92b4-2a186a933d78up+replayingreplaying, {"bytes_per_second":3243622.4,"bytes_per_snapshot":64872448.0,"last_snapshot_bytes":64872448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57image-primary29d6ca525-e53f-4cf8-80fc-35c7b5df9facup+replayingreplaying, {"bytes_per_second":3231334.4,"bytes_per_snapshot":64626688.0,"last_snapshot_bytes":64626688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+replayingreplaying, {"bytes_per_second":3162316.8,"bytes_per_snapshot":63246336.0,"last_snapshot_bytes":63246336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+replayingreplaying, {"bytes_per_second":3237683.2,"bytes_per_snapshot":64753664.0,"last_snapshot_bytes":64753664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+replayingreplaying, {"bytes_per_second":3181158.4,"bytes_per_snapshot":63623168.0,"last_snapshot_bytes":63623168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+replayingreplaying, {"bytes_per_second":3220889.6,"bytes_per_snapshot":64417792.0,"last_snapshot_bytes":64417792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":3354214.4,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620015,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+replayingreplaying, {"bytes_per_second":3345817.6,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+replayingreplaying, {"bytes_per_second":3074252.8,"bytes_per_snapshot":61485056.0,"last_snapshot_bytes":61485056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57' 2024-09-18T00:40:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:40:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-18T00:40:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:40:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-18T00:40:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:40:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:40:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:40:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:40:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-18T00:40:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:40:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:40:57.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-18T00:40:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-18T00:40:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:40:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-18T00:40:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary2 2024-09-18T00:40:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:40:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary2 2024-09-18T00:40:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:40:59.323 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary2 2024-09-18T00:40:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary3 2024-09-18T00:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary3 2024-09-18T00:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-18T00:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:40:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:40:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:40:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:40:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:41:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-18T00:41:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:41:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:41:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:41:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-18T00:41:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary3: 2024-09-18T00:41:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 91db1fb0-ad2a-47fe-bde5-20bf3e90ea37 2024-09-18T00:41:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:41:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3162316.8,"bytes_per_snapshot":63246336.0,"last_snapshot_bytes":63246336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"} 2024-09-18T00:41:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:41:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:47 2024-09-18T00:41:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:41:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:41:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:41:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:41:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:57' 2024-09-18T00:41:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:41:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:41:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:41:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3162316.8,"bytes_per_snapshot":63246336.0,"last_snapshot_bytes":63246336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"} 2024-09-18T00:41:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:41:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:41:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:41:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:41:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:41:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:41:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:41:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-18 00:40:47' 2024-09-18T00:41:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:00.389 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:41:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:47' 2024-09-18T00:41:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:00.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:41:00.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK842634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary101904d784-4580-4531-92b4-2a186a933d78up+replayingreplaying, {"bytes_per_second":3243622.4,"bytes_per_snapshot":64872448.0,"last_snapshot_bytes":64872448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57image-primary391db1fb0-ad2a-47fe-bde5-20bf3e90ea37up+replayingreplaying, {"bytes_per_second":3162316.8,"bytes_per_snapshot":63246336.0,"last_snapshot_bytes":63246336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+replayingreplaying, {"bytes_per_second":3237683.2,"bytes_per_snapshot":64753664.0,"last_snapshot_bytes":64753664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+replayingreplaying, {"bytes_per_second":3181158.4,"bytes_per_snapshot":63623168.0,"last_snapshot_bytes":63623168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+replayingreplaying, {"bytes_per_second":3220889.6,"bytes_per_snapshot":64417792.0,"last_snapshot_bytes":64417792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":3354214.4,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620015,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+replayingreplaying, {"bytes_per_second":3345817.6,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+replayingreplaying, {"bytes_per_second":3074252.8,"bytes_per_snapshot":61485056.0,"last_snapshot_bytes":61485056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:40:57' 2024-09-18T00:41:00.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:41:00.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-18T00:41:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:41:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-18T00:41:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:41:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:41:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-18T00:41:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:41:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:41:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-18T00:41:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-18T00:41:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:41:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-18T00:41:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary3 2024-09-18T00:41:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:41:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:41:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary3 2024-09-18T00:41:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary3 2024-09-18T00:41:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:41:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary4 2024-09-18T00:41:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary4 2024-09-18T00:41:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:41:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:41:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-18T00:41:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:41:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:41:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:41:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:41:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:41:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-18T00:41:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:41:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:41:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:41:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:03.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-18T00:41:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary4: 2024-09-18T00:41:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: a0539cb4-609b-47fd-8ae0-93b0a3b749f0 2024-09-18T00:41:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:41:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3237683.2,"bytes_per_snapshot":64753664.0,"last_snapshot_bytes":64753664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"} 2024-09-18T00:41:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:41:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:47 2024-09-18T00:41:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:41:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:41:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:41:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:41:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:41:02' 2024-09-18T00:41:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:41:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:41:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:41:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3237683.2,"bytes_per_snapshot":64753664.0,"last_snapshot_bytes":64753664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"} 2024-09-18T00:41:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:41:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:41:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:41:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:41:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:41:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:41:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:41:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-18 00:40:47' 2024-09-18T00:41:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:41:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:47' 2024-09-18T00:41:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:41:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK742634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary101904d784-4580-4531-92b4-2a186a933d78up+replayingreplaying, {"bytes_per_second":3243622.4,"bytes_per_snapshot":64872448.0,"last_snapshot_bytes":64872448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:02image-primary4a0539cb4-609b-47fd-8ae0-93b0a3b749f0up+replayingreplaying, {"bytes_per_second":3237683.2,"bytes_per_snapshot":64753664.0,"last_snapshot_bytes":64753664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:02image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+replayingreplaying, {"bytes_per_second":3181158.4,"bytes_per_snapshot":63623168.0,"last_snapshot_bytes":63623168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:02image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+replayingreplaying, {"bytes_per_second":3220889.6,"bytes_per_snapshot":64417792.0,"last_snapshot_bytes":64417792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:02image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":3354214.4,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620015,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:02image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+replayingreplaying, {"bytes_per_second":3345817.6,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:02image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+replayingreplaying, {"bytes_per_second":3074252.8,"bytes_per_snapshot":61485056.0,"last_snapshot_bytes":61485056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:02' 2024-09-18T00:41:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:41:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-18T00:41:03.542 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:41:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-18T00:41:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:41:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:41:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-18T00:41:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:41:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:41:03.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-18T00:41:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-18T00:41:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:41:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-18T00:41:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary4 2024-09-18T00:41:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:41:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:41:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary4 2024-09-18T00:41:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary4 2024-09-18T00:41:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:41:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary5 2024-09-18T00:41:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary5 2024-09-18T00:41:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:41:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:41:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-18T00:41:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:41:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:41:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:41:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:41:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:41:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-18T00:41:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:41:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:41:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:41:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-18T00:41:06.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary5: 2024-09-18T00:41:06.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 6edb0522-635d-46cd-8cf6-76f54f3926a0 2024-09-18T00:41:06.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:41:06.360 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3181158.4,"bytes_per_snapshot":63623168.0,"last_snapshot_bytes":63623168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"} 2024-09-18T00:41:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:41:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:47 2024-09-18T00:41:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:41:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:41:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:41:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:41:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:41:02' 2024-09-18T00:41:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:41:06.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:41:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:41:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3181158.4,"bytes_per_snapshot":63623168.0,"last_snapshot_bytes":63623168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"} 2024-09-18T00:41:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:41:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:41:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:06.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:41:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:41:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:41:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:41:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:41:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-18 00:40:47' 2024-09-18T00:41:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:41:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:47' 2024-09-18T00:41:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:41:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK642634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary101904d784-4580-4531-92b4-2a186a933d78up+replayingreplaying, {"bytes_per_second":3243622.4,"bytes_per_snapshot":64872448.0,"last_snapshot_bytes":64872448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:02image-primary56edb0522-635d-46cd-8cf6-76f54f3926a0up+replayingreplaying, {"bytes_per_second":3181158.4,"bytes_per_snapshot":63623168.0,"last_snapshot_bytes":63623168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:02image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+replayingreplaying, {"bytes_per_second":3220889.6,"bytes_per_snapshot":64417792.0,"last_snapshot_bytes":64417792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:02image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":3354214.4,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620015,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:02image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+replayingreplaying, {"bytes_per_second":3345817.6,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:02image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+replayingreplaying, {"bytes_per_second":3074252.8,"bytes_per_snapshot":61485056.0,"last_snapshot_bytes":61485056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:02' 2024-09-18T00:41:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:41:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-18T00:41:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:41:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-18T00:41:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:41:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:41:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-18T00:41:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:41:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:41:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-18T00:41:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-18T00:41:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:41:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-18T00:41:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary5 2024-09-18T00:41:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:41:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:41:08.305 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary5 2024-09-18T00:41:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary5 2024-09-18T00:41:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:41:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary6 2024-09-18T00:41:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary6 2024-09-18T00:41:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:41:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:41:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-18T00:41:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:41:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:41:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:41:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:41:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:41:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-18T00:41:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:41:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:41:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:41:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:09.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-18T00:41:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary6: 2024-09-18T00:41:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c19919df-d5b3-43ec-abfb-adf02bda5de0 2024-09-18T00:41:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:41:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3220889.6,"bytes_per_snapshot":64417792.0,"last_snapshot_bytes":64417792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"} 2024-09-18T00:41:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:41:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:47 2024-09-18T00:41:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:41:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:41:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:41:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:41:09.396 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:41:07' 2024-09-18T00:41:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:41:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:41:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:41:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3220889.6,"bytes_per_snapshot":64417792.0,"last_snapshot_bytes":64417792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"} 2024-09-18T00:41:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:41:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:09.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:41:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:41:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:41:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:41:09.400 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:41:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:41:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-18 00:40:47' 2024-09-18T00:41:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:41:09.402 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:47' 2024-09-18T00:41:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:41:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK542634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary101904d784-4580-4531-92b4-2a186a933d78up+replayingreplaying, {"bytes_per_second":3243622.4,"bytes_per_snapshot":64872448.0,"last_snapshot_bytes":64872448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:07image-primary6c19919df-d5b3-43ec-abfb-adf02bda5de0up+replayingreplaying, {"bytes_per_second":3220889.6,"bytes_per_snapshot":64417792.0,"last_snapshot_bytes":64417792,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:07image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":3354214.4,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620015,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:07image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+replayingreplaying, {"bytes_per_second":3345817.6,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:07image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+replayingreplaying, {"bytes_per_second":3074252.8,"bytes_per_snapshot":61485056.0,"last_snapshot_bytes":61485056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:07' 2024-09-18T00:41:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:41:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-18T00:41:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:41:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-18T00:41:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:41:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:41:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-18T00:41:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:41:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:41:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-18T00:41:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-18T00:41:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:41:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-18T00:41:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary6 2024-09-18T00:41:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:41:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:41:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary6 2024-09-18T00:41:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary6 2024-09-18T00:41:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:41:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary7 2024-09-18T00:41:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary7 2024-09-18T00:41:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:41:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:41:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-18T00:41:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:41:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:41:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:41:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:41:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:41:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-18T00:41:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:41:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:41:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:41:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-18T00:41:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary7: 2024-09-18T00:41:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: f1fe4e73-72ac-4f20-85b7-52652fb1cf4b 2024-09-18T00:41:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:41:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3354214.4,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620015,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"} 2024-09-18T00:41:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:41:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:47 2024-09-18T00:41:12.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:41:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:41:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:41:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:41:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:41:07' 2024-09-18T00:41:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:41:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:41:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:41:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3354214.4,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620015,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"} 2024-09-18T00:41:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:41:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:41:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:12.370 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:41:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:41:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:41:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:41:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:41:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-18 00:40:47' 2024-09-18T00:41:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:41:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:47' 2024-09-18T00:41:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:41:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK442634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary101904d784-4580-4531-92b4-2a186a933d78up+replayingreplaying, {"bytes_per_second":3243622.4,"bytes_per_snapshot":64872448.0,"last_snapshot_bytes":64872448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:07image-primary7f1fe4e73-72ac-4f20-85b7-52652fb1cf4bup+replayingreplaying, {"bytes_per_second":3354214.4,"bytes_per_snapshot":67084288.0,"last_snapshot_bytes":67084288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620015,"remote_snapshot_timestamp":1726620015,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:12image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+replayingreplaying, {"bytes_per_second":3345817.6,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:12image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+replayingreplaying, {"bytes_per_second":3074252.8,"bytes_per_snapshot":61485056.0,"last_snapshot_bytes":61485056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:07' 2024-09-18T00:41:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:41:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-18T00:41:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:41:12.484 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-18T00:41:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:41:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:12.486 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:41:12.487 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-18T00:41:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:41:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:41:12.488 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-18T00:41:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-18T00:41:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:41:14.254 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-18T00:41:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary7 2024-09-18T00:41:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:41:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:41:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary7 2024-09-18T00:41:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary7 2024-09-18T00:41:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:41:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary8 2024-09-18T00:41:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary8 2024-09-18T00:41:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:41:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:41:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-18T00:41:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:41:14.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:14.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:41:14.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:41:14.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:41:14.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:41:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-18T00:41:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:41:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:41:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:41:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-18T00:41:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary8: 2024-09-18T00:41:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 5dd62c2e-254c-4272-abb3-5adeb3b07fac 2024-09-18T00:41:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:41:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":3345817.6,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"} 2024-09-18T00:41:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:41:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:40:47 2024-09-18T00:41:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:41:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:41:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:41:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:41:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:41:12' 2024-09-18T00:41:15.346 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:41:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:41:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:41:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":3345817.6,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"} 2024-09-18T00:41:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:41:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:41:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:15.349 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:41:15.350 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:41:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:41:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:41:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:41:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-18 00:40:47' 2024-09-18T00:41:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:41:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:40:47' 2024-09-18T00:41:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:41:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK342634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary101904d784-4580-4531-92b4-2a186a933d78up+replayingreplaying, {"bytes_per_second":3243622.4,"bytes_per_snapshot":64872448.0,"last_snapshot_bytes":64872448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:12image-primary85dd62c2e-254c-4272-abb3-5adeb3b07facup+replayingreplaying, {"bytes_per_second":3345817.6,"bytes_per_snapshot":66916352.0,"last_snapshot_bytes":66916352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620019,"remote_snapshot_timestamp":1726620019,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:12image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+replayingreplaying, {"bytes_per_second":3074252.8,"bytes_per_snapshot":61485056.0,"last_snapshot_bytes":61485056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:40:47cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:12' 2024-09-18T00:41:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:41:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-18T00:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:40:47' 2024-09-18T00:41:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-18T00:41:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:41:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:41:15.420 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-18T00:41:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:41:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:41:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-18T00:41:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-18T00:41:17.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:41:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-18T00:41:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary8 2024-09-18T00:41:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:41:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:41:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary8 2024-09-18T00:41:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary8 2024-09-18T00:41:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:41:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary9 2024-09-18T00:41:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary9 2024-09-18T00:41:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:41:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:41:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-18T00:41:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:41:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:41:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:41:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:41:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:41:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-18T00:41:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:41:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:41:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:41:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-18T00:41:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary9: 2024-09-18T00:41:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: c9e2c879-a8a6-4001-8b94-cf4dbb212334 2024-09-18T00:41:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:41:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61485056.0,"last_snapshot_bytes":61485056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"} 2024-09-18T00:41:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:41:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:41:17 2024-09-18T00:41:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:41:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:41:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:41:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:41:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:41:17' 2024-09-18T00:41:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:41:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:41:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:41:18.392 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61485056.0,"last_snapshot_bytes":61485056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"} 2024-09-18T00:41:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:41:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:41:18.393 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:41:18.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:41:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:41:18.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:41:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:41:17' 2024-09-18T00:41:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-18 00:41:17' 2024-09-18T00:41:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:41:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:41:17' 2024-09-18T00:41:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:18.398 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:41:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK242634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary101904d784-4580-4531-92b4-2a186a933d78up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64872448.0,"last_snapshot_bytes":64872448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:41:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:17image-primary9c9e2c879-a8a6-4001-8b94-cf4dbb212334up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":61485056.0,"last_snapshot_bytes":61485056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:41:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:17' 2024-09-18T00:41:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:41:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-18T00:41:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:41:17' 2024-09-18T00:41:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-18T00:41:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:41:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:41:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-18T00:41:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:41:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:41:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-18T00:41:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-18T00:41:20.260 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:41:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-18T00:41:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary9 2024-09-18T00:41:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:41:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:41:20.264 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary9 2024-09-18T00:41:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:20.300 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary9 2024-09-18T00:41:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for j in {1..10} 2024-09-18T00:41:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ IMG=image-primary10 2024-09-18T00:41:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_replaying_status_in_pool_dir cluster1 mirror image-primary10 2024-09-18T00:41:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:41:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' snapshot = journal ']' 2024-09-18T00:41:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-18T00:41:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:41:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:41:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:41:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:41:20.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:41:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-18T00:41:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:41:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local description_pattern= 2024-09-18T00:41:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local service_pattern= 2024-09-18T00:41:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-18T00:41:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='image-primary10: 2024-09-18T00:41:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: global_id: 1904d784-4580-4531-92b4-2a186a933d78 2024-09-18T00:41:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+replaying 2024-09-18T00:41:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64872448.0,"last_snapshot_bytes":64872448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"} 2024-09-18T00:41:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: service: mirror.0 on smithi192 2024-09-18T00:41:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:41:17 2024-09-18T00:41:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: peer_sites: 2024-09-18T00:41:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: name: cluster2 2024-09-18T00:41:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: state: up+stopped 2024-09-18T00:41:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: description: local image is primary 2024-09-18T00:41:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: last_update: 2024-09-18 00:41:17' 2024-09-18T00:41:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ state: .*up+replaying' 2024-09-18T00:41:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: state: up+replaying 2024-09-18T00:41:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^ description: .*' 2024-09-18T00:41:21.363 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64872448.0,"last_snapshot_bytes":64872448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"} 2024-09-18T00:41:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n '' ']' 2024-09-18T00:41:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep '^up+' 2024-09-18T00:41:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep 'service: *mirror..* on ' 2024-09-18T00:41:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout: service: mirror.0 on smithi192 2024-09-18T00:41:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update 2024-09-18T00:41:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-18T00:41:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:41:17' 2024-09-18T00:41:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-18 00:41:17' 2024-09-18T00:41:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:41:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local state_pattern=up+replaying 2024-09-18T00:41:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local 'prev_last_update=2024-09-18 00:41:17' 2024-09-18T00:41:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local status 2024-09-18T00:41:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ CEPH_ARGS= 2024-09-18T00:41:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-18T00:41:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ status='OKOKOK142634266mirror.0smithi19219.1.1-330-gc2ddf972trueOKimage-primary101904d784-4580-4531-92b4-2a186a933d78up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64872448.0,"last_snapshot_bytes":64872448,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726620021,"remote_snapshot_timestamp":1726620021,"replay_state":"idle"}42634266mirror.0smithi1922024-09-18 00:41:17cluster20ec50a57-7d9b-43df-9647-b2d89ea7594dup+stoppedlocal image is primary2024-09-18 00:41:17' 2024-09-18T00:41:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local last_update state 2024-09-18T00:41:21.431 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-18T00:41:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ last_update='2024-09-18 00:41:17' 2024-09-18T00:41:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-18T00:41:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ state=up+replaying 2024-09-18T00:41:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo up+replaying 2024-09-18T00:41:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep up+replaying 2024-09-18T00:41:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:up+replaying 2024-09-18T00:41:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:21.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-18T00:41:21.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:41:21.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:21.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:41:21.436 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-18T00:41:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-18T00:41:23.390 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-18T00:41:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-18T00:41:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pool=mirror 2024-09-18T00:41:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local image=image-primary10 2024-09-18T00:41:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-18T00:41:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-18T00:41:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ grep -wq image-primary10 2024-09-18T00:41:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ return 0 2024-09-18T00:41:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -rf test-primary10 2024-09-18T00:41:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo OK 2024-09-18T00:41:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:OK 2024-09-18T00:41:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ cleanup 0 2024-09-18T00:41:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local error_code=0 2024-09-18T00:41:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ set +e 2024-09-18T00:41:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -ne 0 ']' 2024-09-18T00:41:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -z '' ']' 2024-09-18T00:41:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster instance 2024-09-18T00:41:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-09-18T00:41:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-18T00:41:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror' does not exist 2024-09-18T00:41:24.167 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-09-18T00:41:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-18T00:41:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror' does not exist 2024-09-18T00:41:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-09-18T00:41:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-18T00:41:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror_parent' does not exist 2024-09-18T00:41:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ CEPH_ARGS= 2024-09-18T00:41:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-18T00:41:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:pool 'mirror_parent' does not exist 2024-09-18T00:41:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-18T00:41:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stop_mirrors cluster1 2024-09-18T00:41:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1 2024-09-18T00:41:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local sig= 2024-09-18T00:41:27.269 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 0 2024-09-18T00:41:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T00:41:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stop_mirror cluster1:0 '' 2024-09-18T00:41:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster1:0 2024-09-18T00:41:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local sig= 2024-09-18T00:41:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-09-18T00:41:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pid 2024-09-18T00:41:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_pid_file cluster1:0 2024-09-18T00:41:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster1:0 2024-09-18T00:41:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-09-18T00:41:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T00:41:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster1:0 2024-09-18T00:41:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=cluster 2024-09-18T00:41:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-09-18T00:41:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-09-18T00:41:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-09-18T00:41:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-18T00:41:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval cluster=cluster1 2024-09-18T00:41:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ cluster=cluster1 2024-09-18T00:41:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-09-18T00:41:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-09-18T00:41:27.285 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-18T00:41:27.310 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-18T00:41:27.311 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-18T00:41:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ pid=35617 2024-09-18T00:41:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 35617 ']' 2024-09-18T00:41:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ kill 35617 2024-09-18T00:41:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 16 32 2024-09-18T00:41:27.312 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:41:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ps auxww 2024-09-18T00:41:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ awk -v pid=35617 '$2 == pid {print; exit 1}' 2024-09-18T00:41:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-09-18T00:41:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ps auxww 2024-09-18T00:41:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ awk -v pid=35617 '$2 == pid {print; exit 1}' 2024-09-18T00:41:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-18T00:41:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-09-18T00:41:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-09-18T00:41:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-09-18T00:41:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T00:41:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-09-18T00:41:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-09-18T00:41:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-09-18T00:41:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-09-18T00:41:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-09-18T00:41:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T00:41:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-09-18T00:41:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-09-18T00:41:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-09-18T00:41:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-09-18T00:41:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T00:41:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T00:41:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T00:41:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-09-18T00:41:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster1:0 2024-09-18T00:41:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-09-18T00:41:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-09-18T00:41:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-18T00:41:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-09-18T00:41:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-09-18T00:41:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-09-18T00:41:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-09-18T00:41:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-09-18T00:41:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T00:41:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster1 2024-09-18T00:41:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster1 2024-09-18T00:41:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-09-18T00:41:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-09-18T00:41:28.382 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-18T00:41:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T00:41:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-18T00:41:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_pid_file cluster1:0 2024-09-18T00:41:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1:0 2024-09-18T00:41:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-09-18T00:41:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-18T00:41:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster1:0 2024-09-18T00:41:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-09-18T00:41:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-09-18T00:41:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster1 2024-09-18T00:41:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-09-18T00:41:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-18T00:41:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster1 2024-09-18T00:41:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster1 2024-09-18T00:41:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-09-18T00:41:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-09-18T00:41:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-18T00:41:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-18T00:41:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-18T00:41:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-18T00:41:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stop_mirrors cluster2 2024-09-18T00:41:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2 2024-09-18T00:41:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local sig= 2024-09-18T00:41:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ seq 0 0 2024-09-18T00:41:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-18T00:41:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ stop_mirror cluster2:0 '' 2024-09-18T00:41:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local cluster=cluster2:0 2024-09-18T00:41:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local sig= 2024-09-18T00:41:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test -n '' 2024-09-18T00:41:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ local pid 2024-09-18T00:41:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ daemon_pid_file cluster2:0 2024-09-18T00:41:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster=cluster2:0 2024-09-18T00:41:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance 2024-09-18T00:41:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2024-09-18T00:41:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local val=cluster2:0 2024-09-18T00:41:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local cluster_var_name=cluster 2024-09-18T00:41:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local instance_var_name=instance 2024-09-18T00:41:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster2 2024-09-18T00:41:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-09-18T00:41:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-18T00:41:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval cluster=cluster2 2024-09-18T00:41:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ cluster=cluster2 2024-09-18T00:41:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ eval instance=0 2024-09-18T00:41:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ instance=0 2024-09-18T00:41:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-18T00:41:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-18T00:41:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-18T00:41:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ pid=35653 2024-09-18T00:41:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -n 35653 ']' 2024-09-18T00:41:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ kill 35653 2024-09-18T00:41:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ for s in 1 2 4 8 16 32 2024-09-18T00:41:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ sleep 1 2024-09-18T00:41:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ps auxww 2024-09-18T00:41:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ awk -v pid=35653 '$2 == pid {print; exit 1}' 2024-09-18T00:41:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ break 2024-09-18T00:41:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ ps auxww 2024-09-18T00:41:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ awk -v pid=35653 '$2 == pid {print; exit 1}' 2024-09-18T00:41:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster2:0 cluster1 2024-09-18T00:41:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster2:0 2024-09-18T00:41:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster1 2024-09-18T00:41:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-09-18T00:41:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-18T00:41:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster2:0 2024-09-18T00:41:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-09-18T00:41:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-09-18T00:41:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster2 2024-09-18T00:41:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-09-18T00:41:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-18T00:41:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster2 2024-09-18T00:41:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster2 2024-09-18T00:41:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-09-18T00:41:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-09-18T00:41:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-18T00:41:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-18T00:41:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-18T00:41:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-18T00:41:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local local_cluster=cluster2:0 2024-09-18T00:41:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2 2024-09-18T00:41:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-09-18T00:41:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-18T00:41:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster2:0 2024-09-18T00:41:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=local_cluster 2024-09-18T00:41:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-09-18T00:41:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster2 2024-09-18T00:41:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-09-18T00:41:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-18T00:41:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval local_cluster=cluster2 2024-09-18T00:41:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ local_cluster=cluster2 2024-09-18T00:41:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-09-18T00:41:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-09-18T00:41:29.505 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-18T00:41:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-18T00:41:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-18T00:41:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ daemon_pid_file cluster2:0 2024-09-18T00:41:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster=cluster2:0 2024-09-18T00:41:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance 2024-09-18T00:41:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-18T00:41:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local val=cluster2:0 2024-09-18T00:41:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local cluster_var_name=cluster 2024-09-18T00:41:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ local instance_var_name=instance 2024-09-18T00:41:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ cluster=cluster2 2024-09-18T00:41:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ instance=0 2024-09-18T00:41:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-18T00:41:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval cluster=cluster2 2024-09-18T00:41:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ cluster=cluster2 2024-09-18T00:41:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ eval instance=0 2024-09-18T00:41:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ instance=0 2024-09-18T00:41:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-18T00:41:29.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-18T00:41:29.546 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-18T00:41:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' -z 1 ']' 2024-09-18T00:41:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ test '' = /tmp/tmp.LnAXrU1SbO 2024-09-18T00:41:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ rm -Rf /tmp/tmp.LnAXrU1SbO 2024-09-18T00:41:29.548 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stdout:OK 2024-09-18T00:41:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ '[' 0 -eq 0 ']' 2024-09-18T00:41:29.549 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ echo OK 2024-09-18T00:41:29.550 INFO:tasks.workunit.cluster1.client.mirror.smithi192.stderr:+ exit 0 2024-09-18T00:41:29.550 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-18T00:41:29.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-18T00:41:29.768 INFO:tasks.workunit:Stopping ['rbd/compare_mirror_images.sh'] on cluster1.client.mirror... 2024-09-18T00:41:29.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-18T00:41:30.261 DEBUG:teuthology.parallel:result is None 2024-09-18T00:41:30.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-18T00:41:30.286 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-18T00:41:30.286 DEBUG:teuthology.orchestra.run.smithi192:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-18T00:41:30.342 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-18T00:41:30.342 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-18T00:41:30.359 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-18T00:41:30.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-18T00:41:30.683 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-18T00:41:30.683 INFO:teuthology.orchestra.run.smithi192.stderr:dumped all 2024-09-18T00:41:30.695 INFO:teuthology.orchestra.run.smithi192.stdout:{"pg_ready":true,"pg_map":{"version":3947,"stamp":"2024-09-18T00:41:28.787058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":130,"num_read_kb":113,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":227,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2544620,"kb_used_data":150704,"kb_used_omap":4,"kb_used_meta":2389755,"kb_avail":280570900,"statfs":{"total":289910292480,"available":287304601600,"internally_reserved":0,"allocated":154320896,"data_stored":233102541,"data_compressed":18072975,"data_compressed_allocated":21901312,"data_compressed_original":107806720,"omap_allocated":4785,"internal_metadata":2447109455},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-616119840,"num_objects":-233,"num_object_clones":-21,"num_object_copies":-466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-233,"num_whiteouts":0,"num_read":-708634,"num_read_kb":-36664036,"num_write":-148238,"num_write_kb":-29404580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-9,"num_bytes_recovered":-8450722,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001274"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3701,"reported_epoch":1840,"state":"active+clean","last_fresh":"2024-09-18T00:41:23.134788+0000","last_change":"2024-09-17T23:31:23.507145+0000","last_active":"2024-09-18T00:41:23.134788+0000","last_peered":"2024-09-18T00:41:23.134788+0000","last_clean":"2024-09-18T00:41:23.134788+0000","last_became_active":"2024-09-17T23:31:21.498864+0000","last_became_peered":"2024-09-17T23:31:21.498864+0000","last_unstale":"2024-09-18T00:41:23.134788+0000","last_undegraded":"2024-09-18T00:41:23.134788+0000","last_fullsized":"2024-09-18T00:41:23.134788+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:08:07.786924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083165799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3701,"reported_epoch":1840,"state":"active+clean","last_fresh":"2024-09-18T00:41:23.134682+0000","last_change":"2024-09-17T23:31:23.507426+0000","last_active":"2024-09-18T00:41:23.134682+0000","last_peered":"2024-09-18T00:41:23.134682+0000","last_clean":"2024-09-18T00:41:23.134682+0000","last_became_active":"2024-09-17T23:31:21.498384+0000","last_became_peered":"2024-09-17T23:31:21.498384+0000","last_unstale":"2024-09-18T00:41:23.134682+0000","last_undegraded":"2024-09-18T00:41:23.134682+0000","last_fullsized":"2024-09-18T00:41:23.134682+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:36:11.256952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010814310000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3702,"reported_epoch":1840,"state":"active+clean","last_fresh":"2024-09-18T00:41:23.134048+0000","last_change":"2024-09-17T23:31:23.506777+0000","last_active":"2024-09-18T00:41:23.134048+0000","last_peered":"2024-09-18T00:41:23.134048+0000","last_clean":"2024-09-18T00:41:23.134048+0000","last_became_active":"2024-09-17T23:31:21.497851+0000","last_became_peered":"2024-09-17T23:31:21.497851+0000","last_unstale":"2024-09-18T00:41:23.134048+0000","last_undegraded":"2024-09-18T00:41:23.134048+0000","last_fullsized":"2024-09-18T00:41:23.134048+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:30:54.399154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049346500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3702,"reported_epoch":1840,"state":"active+clean","last_fresh":"2024-09-18T00:41:23.132930+0000","last_change":"2024-09-17T23:31:23.507092+0000","last_active":"2024-09-18T00:41:23.132930+0000","last_peered":"2024-09-18T00:41:23.132930+0000","last_clean":"2024-09-18T00:41:23.132930+0000","last_became_active":"2024-09-17T23:31:21.497623+0000","last_became_peered":"2024-09-17T23:31:21.497623+0000","last_unstale":"2024-09-18T00:41:23.132930+0000","last_undegraded":"2024-09-18T00:41:23.132930+0000","last_fullsized":"2024-09-18T00:41:23.132930+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:32:29.760120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079303000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":3707,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.512984+0000","last_change":"2024-09-17T23:31:23.511202+0000","last_active":"2024-09-18T00:41:27.512984+0000","last_peered":"2024-09-18T00:41:27.512984+0000","last_clean":"2024-09-18T00:41:27.512984+0000","last_became_active":"2024-09-17T23:31:21.498502+0000","last_became_peered":"2024-09-17T23:31:21.498502+0000","last_unstale":"2024-09-18T00:41:27.512984+0000","last_undegraded":"2024-09-18T00:41:27.512984+0000","last_fullsized":"2024-09-18T00:41:27.512984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:08:28.112030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094839100000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3702,"reported_epoch":1840,"state":"active+clean","last_fresh":"2024-09-18T00:41:23.129987+0000","last_change":"2024-09-17T23:31:24.091376+0000","last_active":"2024-09-18T00:41:23.129987+0000","last_peered":"2024-09-18T00:41:23.129987+0000","last_clean":"2024-09-18T00:41:23.129987+0000","last_became_active":"2024-09-17T23:31:21.496953+0000","last_became_peered":"2024-09-17T23:31:21.496953+0000","last_unstale":"2024-09-18T00:41:23.129987+0000","last_undegraded":"2024-09-18T00:41:23.129987+0000","last_fullsized":"2024-09-18T00:41:23.129987+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:40:43.130208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031250800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3703,"reported_epoch":1840,"state":"active+clean","last_fresh":"2024-09-18T00:41:23.129291+0000","last_change":"2024-09-17T23:31:24.091323+0000","last_active":"2024-09-18T00:41:23.129291+0000","last_peered":"2024-09-18T00:41:23.129291+0000","last_clean":"2024-09-18T00:41:23.129291+0000","last_became_active":"2024-09-17T23:31:21.497936+0000","last_became_peered":"2024-09-17T23:31:21.497936+0000","last_unstale":"2024-09-18T00:41:23.129291+0000","last_undegraded":"2024-09-18T00:41:23.129291+0000","last_fullsized":"2024-09-18T00:41:23.129291+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:22:43.988917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030225,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":3703,"reported_epoch":1840,"state":"active+clean","last_fresh":"2024-09-18T00:41:23.133773+0000","last_change":"2024-09-17T23:31:23.506776+0000","last_active":"2024-09-18T00:41:23.133773+0000","last_peered":"2024-09-18T00:41:23.133773+0000","last_clean":"2024-09-18T00:41:23.133773+0000","last_became_active":"2024-09-17T23:31:21.497054+0000","last_became_peered":"2024-09-17T23:31:21.497054+0000","last_unstale":"2024-09-18T00:41:23.133773+0000","last_undegraded":"2024-09-18T00:41:23.133773+0000","last_fullsized":"2024-09-18T00:41:23.133773+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:43:27.049283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051159800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"813'125","reported_seq":3866,"reported_epoch":1840,"state":"active+clean","last_fresh":"2024-09-18T00:41:23.134085+0000","last_change":"2024-09-17T23:31:15.637809+0000","last_active":"2024-09-18T00:41:23.134085+0000","last_peered":"2024-09-18T00:41:23.134085+0000","last_clean":"2024-09-18T00:41:23.134085+0000","last_became_active":"2024-09-17T23:31:15.636714+0000","last_became_peered":"2024-09-17T23:31:15.636714+0000","last_unstale":"2024-09-18T00:41:23.134085+0000","last_undegraded":"2024-09-18T00:41:23.134085+0000","last_fullsized":"2024-09-18T00:41:23.134085+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:14.466759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:14.466759+0000","last_clean_scrub_stamp":"2024-09-17T23:31:14.466759+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:05:04.207207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":113,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":113,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6090,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706512,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":784616,"kb_used_data":5200,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":93587224,"statfs":{"total":96636764160,"available":95833317376,"internally_reserved":0,"allocated":5324800,"data_stored":2956430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739215,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":857252,"kb_used_data":140240,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":93514588,"statfs":{"total":96636764160,"available":95758938112,"internally_reserved":0,"allocated":143605760,"data_stored":226595168,"data_compressed":18069930,"data_compressed_allocated":21864448,"data_compressed_original":107233280,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739214,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":902752,"kb_used_data":5264,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":93469088,"statfs":{"total":96636764160,"available":95712346112,"internally_reserved":0,"allocated":5390336,"data_stored":3550943,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T00:41:30.696 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-18T00:41:30.983 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-18T00:41:30.983 INFO:teuthology.orchestra.run.smithi192.stderr:dumped all 2024-09-18T00:41:30.996 INFO:teuthology.orchestra.run.smithi192.stdout:{"pg_ready":true,"pg_map":{"version":3948,"stamp":"2024-09-18T00:41:30.787395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":130,"num_read_kb":113,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2445828,"kb_used_data":15756,"kb_used_omap":4,"kb_used_meta":2430011,"kb_avail":280669692,"statfs":{"total":289910292480,"available":287405764608,"internally_reserved":0,"allocated":16134144,"data_stored":10069822,"data_compressed":6090,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":2488331599},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-512497809,"num_objects":-191,"num_object_clones":-13,"num_object_copies":-382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-191,"num_whiteouts":0,"num_read":-708677,"num_read_kb":-36664068,"num_write":-148287,"num_write_kb":-29404588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-9,"num_bytes_recovered":-8450722,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.660859"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3705,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.153199+0000","last_change":"2024-09-17T23:31:23.507145+0000","last_active":"2024-09-18T00:41:27.153199+0000","last_peered":"2024-09-18T00:41:27.153199+0000","last_clean":"2024-09-18T00:41:27.153199+0000","last_became_active":"2024-09-17T23:31:21.498864+0000","last_became_peered":"2024-09-17T23:31:21.498864+0000","last_unstale":"2024-09-18T00:41:27.153199+0000","last_undegraded":"2024-09-18T00:41:27.153199+0000","last_fullsized":"2024-09-18T00:41:27.153199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:08:07.786924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083165799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3705,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.153911+0000","last_change":"2024-09-17T23:31:23.507426+0000","last_active":"2024-09-18T00:41:27.153911+0000","last_peered":"2024-09-18T00:41:27.153911+0000","last_clean":"2024-09-18T00:41:27.153911+0000","last_became_active":"2024-09-17T23:31:21.498384+0000","last_became_peered":"2024-09-17T23:31:21.498384+0000","last_unstale":"2024-09-18T00:41:27.153911+0000","last_undegraded":"2024-09-18T00:41:27.153911+0000","last_fullsized":"2024-09-18T00:41:27.153911+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:36:11.256952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010814310000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3706,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.152403+0000","last_change":"2024-09-17T23:31:23.506777+0000","last_active":"2024-09-18T00:41:27.152403+0000","last_peered":"2024-09-18T00:41:27.152403+0000","last_clean":"2024-09-18T00:41:27.152403+0000","last_became_active":"2024-09-17T23:31:21.497851+0000","last_became_peered":"2024-09-17T23:31:21.497851+0000","last_unstale":"2024-09-18T00:41:27.152403+0000","last_undegraded":"2024-09-18T00:41:27.152403+0000","last_fullsized":"2024-09-18T00:41:27.152403+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:30:54.399154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049346500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3706,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.153130+0000","last_change":"2024-09-17T23:31:23.507092+0000","last_active":"2024-09-18T00:41:27.153130+0000","last_peered":"2024-09-18T00:41:27.153130+0000","last_clean":"2024-09-18T00:41:27.153130+0000","last_became_active":"2024-09-17T23:31:21.497623+0000","last_became_peered":"2024-09-17T23:31:21.497623+0000","last_unstale":"2024-09-18T00:41:27.153130+0000","last_undegraded":"2024-09-18T00:41:27.153130+0000","last_fullsized":"2024-09-18T00:41:27.153130+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:32:29.760120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079303000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":3707,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.512984+0000","last_change":"2024-09-17T23:31:23.511202+0000","last_active":"2024-09-18T00:41:27.512984+0000","last_peered":"2024-09-18T00:41:27.512984+0000","last_clean":"2024-09-18T00:41:27.512984+0000","last_became_active":"2024-09-17T23:31:21.498502+0000","last_became_peered":"2024-09-17T23:31:21.498502+0000","last_unstale":"2024-09-18T00:41:27.512984+0000","last_undegraded":"2024-09-18T00:41:27.512984+0000","last_fullsized":"2024-09-18T00:41:27.512984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:08:28.112030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094839100000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3706,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:29.810182+0000","last_change":"2024-09-17T23:31:24.091376+0000","last_active":"2024-09-18T00:41:29.810182+0000","last_peered":"2024-09-18T00:41:29.810182+0000","last_clean":"2024-09-18T00:41:29.810182+0000","last_became_active":"2024-09-17T23:31:21.496953+0000","last_became_peered":"2024-09-17T23:31:21.496953+0000","last_unstale":"2024-09-18T00:41:29.810182+0000","last_undegraded":"2024-09-18T00:41:29.810182+0000","last_fullsized":"2024-09-18T00:41:29.810182+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:40:43.130208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031250800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3707,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:29.810182+0000","last_change":"2024-09-17T23:31:24.091323+0000","last_active":"2024-09-18T00:41:29.810182+0000","last_peered":"2024-09-18T00:41:29.810182+0000","last_clean":"2024-09-18T00:41:29.810182+0000","last_became_active":"2024-09-17T23:31:21.497936+0000","last_became_peered":"2024-09-17T23:31:21.497936+0000","last_unstale":"2024-09-18T00:41:29.810182+0000","last_undegraded":"2024-09-18T00:41:29.810182+0000","last_fullsized":"2024-09-18T00:41:29.810182+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:22:43.988917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030225,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":3707,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.150484+0000","last_change":"2024-09-17T23:31:23.506776+0000","last_active":"2024-09-18T00:41:27.150484+0000","last_peered":"2024-09-18T00:41:27.150484+0000","last_clean":"2024-09-18T00:41:27.150484+0000","last_became_active":"2024-09-17T23:31:21.497054+0000","last_became_peered":"2024-09-17T23:31:21.497054+0000","last_unstale":"2024-09-18T00:41:27.150484+0000","last_undegraded":"2024-09-18T00:41:27.150484+0000","last_fullsized":"2024-09-18T00:41:27.150484+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:43:27.049283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051159800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"813'125","reported_seq":3871,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.149981+0000","last_change":"2024-09-17T23:31:15.637809+0000","last_active":"2024-09-18T00:41:27.149981+0000","last_peered":"2024-09-18T00:41:27.149981+0000","last_clean":"2024-09-18T00:41:27.149981+0000","last_became_active":"2024-09-17T23:31:15.636714+0000","last_became_peered":"2024-09-17T23:31:15.636714+0000","last_unstale":"2024-09-18T00:41:27.149981+0000","last_undegraded":"2024-09-18T00:41:27.149981+0000","last_fullsized":"2024-09-18T00:41:27.149981+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:14.466759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:14.466759+0000","last_clean_scrub_stamp":"2024-09-17T23:31:14.466759+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:05:04.207207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":113,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":113,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6090,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706513,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":816628,"kb_used_data":5216,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":93555212,"statfs":{"total":96636764160,"available":95800537088,"internally_reserved":0,"allocated":5341184,"data_stored":2964246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739216,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726448,"kb_used_data":5276,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93645392,"statfs":{"total":96636764160,"available":95892881408,"internally_reserved":0,"allocated":5402624,"data_stored":3554633,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739214,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":902752,"kb_used_data":5264,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":93469088,"statfs":{"total":96636764160,"available":95712346112,"internally_reserved":0,"allocated":5390336,"data_stored":3550943,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T00:41:30.997 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-18T00:41:30.997 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-18T00:41:31.280 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-18T00:41:31.280 INFO:teuthology.orchestra.run.smithi192.stderr:dumped all 2024-09-18T00:41:31.292 INFO:teuthology.orchestra.run.smithi192.stdout:{"pg_ready":true,"pg_map":{"version":3948,"stamp":"2024-09-18T00:41:30.787395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":130,"num_read_kb":113,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2445828,"kb_used_data":15756,"kb_used_omap":4,"kb_used_meta":2430011,"kb_avail":280669692,"statfs":{"total":289910292480,"available":287405764608,"internally_reserved":0,"allocated":16134144,"data_stored":10069822,"data_compressed":6090,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":2488331599},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-512497809,"num_objects":-191,"num_object_clones":-13,"num_object_copies":-382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-191,"num_whiteouts":0,"num_read":-708677,"num_read_kb":-36664068,"num_write":-148287,"num_write_kb":-29404588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-9,"num_bytes_recovered":-8450722,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.660859"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3705,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.153199+0000","last_change":"2024-09-17T23:31:23.507145+0000","last_active":"2024-09-18T00:41:27.153199+0000","last_peered":"2024-09-18T00:41:27.153199+0000","last_clean":"2024-09-18T00:41:27.153199+0000","last_became_active":"2024-09-17T23:31:21.498864+0000","last_became_peered":"2024-09-17T23:31:21.498864+0000","last_unstale":"2024-09-18T00:41:27.153199+0000","last_undegraded":"2024-09-18T00:41:27.153199+0000","last_fullsized":"2024-09-18T00:41:27.153199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:08:07.786924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083165799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3705,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.153911+0000","last_change":"2024-09-17T23:31:23.507426+0000","last_active":"2024-09-18T00:41:27.153911+0000","last_peered":"2024-09-18T00:41:27.153911+0000","last_clean":"2024-09-18T00:41:27.153911+0000","last_became_active":"2024-09-17T23:31:21.498384+0000","last_became_peered":"2024-09-17T23:31:21.498384+0000","last_unstale":"2024-09-18T00:41:27.153911+0000","last_undegraded":"2024-09-18T00:41:27.153911+0000","last_fullsized":"2024-09-18T00:41:27.153911+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:36:11.256952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010814310000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3706,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.152403+0000","last_change":"2024-09-17T23:31:23.506777+0000","last_active":"2024-09-18T00:41:27.152403+0000","last_peered":"2024-09-18T00:41:27.152403+0000","last_clean":"2024-09-18T00:41:27.152403+0000","last_became_active":"2024-09-17T23:31:21.497851+0000","last_became_peered":"2024-09-17T23:31:21.497851+0000","last_unstale":"2024-09-18T00:41:27.152403+0000","last_undegraded":"2024-09-18T00:41:27.152403+0000","last_fullsized":"2024-09-18T00:41:27.152403+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:30:54.399154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049346500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3706,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.153130+0000","last_change":"2024-09-17T23:31:23.507092+0000","last_active":"2024-09-18T00:41:27.153130+0000","last_peered":"2024-09-18T00:41:27.153130+0000","last_clean":"2024-09-18T00:41:27.153130+0000","last_became_active":"2024-09-17T23:31:21.497623+0000","last_became_peered":"2024-09-17T23:31:21.497623+0000","last_unstale":"2024-09-18T00:41:27.153130+0000","last_undegraded":"2024-09-18T00:41:27.153130+0000","last_fullsized":"2024-09-18T00:41:27.153130+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:32:29.760120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079303000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":3707,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.512984+0000","last_change":"2024-09-17T23:31:23.511202+0000","last_active":"2024-09-18T00:41:27.512984+0000","last_peered":"2024-09-18T00:41:27.512984+0000","last_clean":"2024-09-18T00:41:27.512984+0000","last_became_active":"2024-09-17T23:31:21.498502+0000","last_became_peered":"2024-09-17T23:31:21.498502+0000","last_unstale":"2024-09-18T00:41:27.512984+0000","last_undegraded":"2024-09-18T00:41:27.512984+0000","last_fullsized":"2024-09-18T00:41:27.512984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:08:28.112030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094839100000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3706,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:29.810182+0000","last_change":"2024-09-17T23:31:24.091376+0000","last_active":"2024-09-18T00:41:29.810182+0000","last_peered":"2024-09-18T00:41:29.810182+0000","last_clean":"2024-09-18T00:41:29.810182+0000","last_became_active":"2024-09-17T23:31:21.496953+0000","last_became_peered":"2024-09-17T23:31:21.496953+0000","last_unstale":"2024-09-18T00:41:29.810182+0000","last_undegraded":"2024-09-18T00:41:29.810182+0000","last_fullsized":"2024-09-18T00:41:29.810182+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:40:43.130208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031250800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3707,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:29.810182+0000","last_change":"2024-09-17T23:31:24.091323+0000","last_active":"2024-09-18T00:41:29.810182+0000","last_peered":"2024-09-18T00:41:29.810182+0000","last_clean":"2024-09-18T00:41:29.810182+0000","last_became_active":"2024-09-17T23:31:21.497936+0000","last_became_peered":"2024-09-17T23:31:21.497936+0000","last_unstale":"2024-09-18T00:41:29.810182+0000","last_undegraded":"2024-09-18T00:41:29.810182+0000","last_fullsized":"2024-09-18T00:41:29.810182+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:22:43.988917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030225,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":3707,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.150484+0000","last_change":"2024-09-17T23:31:23.506776+0000","last_active":"2024-09-18T00:41:27.150484+0000","last_peered":"2024-09-18T00:41:27.150484+0000","last_clean":"2024-09-18T00:41:27.150484+0000","last_became_active":"2024-09-17T23:31:21.497054+0000","last_became_peered":"2024-09-17T23:31:21.497054+0000","last_unstale":"2024-09-18T00:41:27.150484+0000","last_undegraded":"2024-09-18T00:41:27.150484+0000","last_fullsized":"2024-09-18T00:41:27.150484+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:43:27.049283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051159800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"813'125","reported_seq":3871,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.149981+0000","last_change":"2024-09-17T23:31:15.637809+0000","last_active":"2024-09-18T00:41:27.149981+0000","last_peered":"2024-09-18T00:41:27.149981+0000","last_clean":"2024-09-18T00:41:27.149981+0000","last_became_active":"2024-09-17T23:31:15.636714+0000","last_became_peered":"2024-09-17T23:31:15.636714+0000","last_unstale":"2024-09-18T00:41:27.149981+0000","last_undegraded":"2024-09-18T00:41:27.149981+0000","last_fullsized":"2024-09-18T00:41:27.149981+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:14.466759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:14.466759+0000","last_clean_scrub_stamp":"2024-09-17T23:31:14.466759+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:05:04.207207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":113,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":113,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6090,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706513,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":816628,"kb_used_data":5216,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":93555212,"statfs":{"total":96636764160,"available":95800537088,"internally_reserved":0,"allocated":5341184,"data_stored":2964246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739216,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726448,"kb_used_data":5276,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93645392,"statfs":{"total":96636764160,"available":95892881408,"internally_reserved":0,"allocated":5402624,"data_stored":3554633,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739214,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":902752,"kb_used_data":5264,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":93469088,"statfs":{"total":96636764160,"available":95712346112,"internally_reserved":0,"allocated":5390336,"data_stored":3550943,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T00:41:31.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-18T00:41:31.576 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-18T00:41:31.576 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1842,"fsid":"d61484e8-a8cb-4030-b6c2-cef28a296bff","created":"2024-09-17T23:31:04.453616+0000","modified":"2024-09-18T00:41:27.135861+0000","last_up_change":"2024-09-17T23:31:14.466759+0000","last_in_change":"2024-09-17T23:31:05.288256+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:31:13.785771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T23:31:19.553055+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dc15472a-cb32-4dcf-a0df-76086c7fed31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6809","nonce":1545791238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6811","nonce":1545791238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6815","nonce":1545791238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":1545791238},{"type":"v1","addr":"172.21.15.192:6813","nonce":1545791238}]},"public_addr":"172.21.15.192:6809/1545791238","cluster_addr":"172.21.15.192:6811/1545791238","heartbeat_back_addr":"172.21.15.192:6815/1545791238","heartbeat_front_addr":"172.21.15.192:6813/1545791238","state":["exists","up"]},{"osd":1,"uuid":"4c9ba9c3-191f-4414-aa4b-46c71c7e21f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6801","nonce":3313808101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6803","nonce":3313808101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6807","nonce":3313808101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":3313808101},{"type":"v1","addr":"172.21.15.192:6805","nonce":3313808101}]},"public_addr":"172.21.15.192:6801/3313808101","cluster_addr":"172.21.15.192:6803/3313808101","heartbeat_back_addr":"172.21.15.192:6807/3313808101","heartbeat_front_addr":"172.21.15.192:6805/3313808101","state":["exists","up"]},{"osd":2,"uuid":"7d465e64-5b7a-4189-bcce-b13ccb458643","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6817","nonce":1728261310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6819","nonce":1728261310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6823","nonce":1728261310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1728261310},{"type":"v1","addr":"172.21.15.192:6821","nonce":1728261310}]},"public_addr":"172.21.15.192:6817/1728261310","cluster_addr":"172.21.15.192:6819/1728261310","heartbeat_back_addr":"172.21.15.192:6823/1728261310","heartbeat_front_addr":"172.21.15.192:6821/1728261310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:31:11.739048+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:31:11.591739+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:31:12.300977+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":3294,"length":1},{"begin":3299,"length":4},{"begin":3309,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T00:41:32.589 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-18T00:41:32.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-18T00:41:32.714 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-18T00:41:32.715 INFO:teuthology.orchestra.run.smithi192.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T00:41:32.715 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-18T00:41:32.725 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-18T00:41:33.003 INFO:teuthology.orchestra.run.smithi192.stderr:instructed osd(s) 0 to deep-scrub 2024-09-18T00:41:33.016 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-18T00:41:33.017 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-18T00:41:33.138 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-18T00:41:33.138 INFO:teuthology.orchestra.run.smithi192.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T00:41:33.138 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-18T00:41:33.149 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-18T00:41:33.425 INFO:teuthology.orchestra.run.smithi192.stderr:instructed osd(s) 1 to deep-scrub 2024-09-18T00:41:33.438 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-18T00:41:33.438 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-18T00:41:33.561 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-09-18T00:41:33.561 INFO:teuthology.orchestra.run.smithi192.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T00:41:33.561 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-09-18T00:41:33.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-18T00:41:33.848 INFO:teuthology.orchestra.run.smithi192.stderr:instructed osd(s) 2 to deep-scrub 2024-09-18T00:41:33.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-18T00:41:34.136 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-18T00:41:34.137 INFO:teuthology.orchestra.run.smithi192.stderr:dumped all 2024-09-18T00:41:34.149 INFO:teuthology.orchestra.run.smithi192.stdout:{"pg_ready":true,"pg_map":{"version":3949,"stamp":"2024-09-18T00:41:32.787779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":130,"num_read_kb":113,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2445828,"kb_used_data":15756,"kb_used_omap":4,"kb_used_meta":2430011,"kb_avail":280669692,"statfs":{"total":289910292480,"available":287405764608,"internally_reserved":0,"allocated":16134144,"data_stored":10069822,"data_compressed":6090,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":2488331599},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-313127640,"num_objects":-120,"num_object_clones":-2,"num_object_copies":-240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-120,"num_whiteouts":0,"num_read":-708763,"num_read_kb":-36664144,"num_write":-148374,"num_write_kb":-29404598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-9,"num_bytes_recovered":-8450722,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001592"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3705,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.153199+0000","last_change":"2024-09-17T23:31:23.507145+0000","last_active":"2024-09-18T00:41:27.153199+0000","last_peered":"2024-09-18T00:41:27.153199+0000","last_clean":"2024-09-18T00:41:27.153199+0000","last_became_active":"2024-09-17T23:31:21.498864+0000","last_became_peered":"2024-09-17T23:31:21.498864+0000","last_unstale":"2024-09-18T00:41:27.153199+0000","last_undegraded":"2024-09-18T00:41:27.153199+0000","last_fullsized":"2024-09-18T00:41:27.153199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:08:07.786924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083165799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3705,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.153911+0000","last_change":"2024-09-17T23:31:23.507426+0000","last_active":"2024-09-18T00:41:27.153911+0000","last_peered":"2024-09-18T00:41:27.153911+0000","last_clean":"2024-09-18T00:41:27.153911+0000","last_became_active":"2024-09-17T23:31:21.498384+0000","last_became_peered":"2024-09-17T23:31:21.498384+0000","last_unstale":"2024-09-18T00:41:27.153911+0000","last_undegraded":"2024-09-18T00:41:27.153911+0000","last_fullsized":"2024-09-18T00:41:27.153911+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T10:36:11.256952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010814310000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3706,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.152403+0000","last_change":"2024-09-17T23:31:23.506777+0000","last_active":"2024-09-18T00:41:27.152403+0000","last_peered":"2024-09-18T00:41:27.152403+0000","last_clean":"2024-09-18T00:41:27.152403+0000","last_became_active":"2024-09-17T23:31:21.497851+0000","last_became_peered":"2024-09-17T23:31:21.497851+0000","last_unstale":"2024-09-18T00:41:27.152403+0000","last_undegraded":"2024-09-18T00:41:27.152403+0000","last_fullsized":"2024-09-18T00:41:27.152403+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T11:30:54.399154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049346500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3706,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.153130+0000","last_change":"2024-09-17T23:31:23.507092+0000","last_active":"2024-09-18T00:41:27.153130+0000","last_peered":"2024-09-18T00:41:27.153130+0000","last_clean":"2024-09-18T00:41:27.153130+0000","last_became_active":"2024-09-17T23:31:21.497623+0000","last_became_peered":"2024-09-17T23:31:21.497623+0000","last_unstale":"2024-09-18T00:41:27.153130+0000","last_undegraded":"2024-09-18T00:41:27.153130+0000","last_fullsized":"2024-09-18T00:41:27.153130+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:32:29.760120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079303000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":3707,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.512984+0000","last_change":"2024-09-17T23:31:23.511202+0000","last_active":"2024-09-18T00:41:27.512984+0000","last_peered":"2024-09-18T00:41:27.512984+0000","last_clean":"2024-09-18T00:41:27.512984+0000","last_became_active":"2024-09-17T23:31:21.498502+0000","last_became_peered":"2024-09-17T23:31:21.498502+0000","last_unstale":"2024-09-18T00:41:27.512984+0000","last_undegraded":"2024-09-18T00:41:27.512984+0000","last_fullsized":"2024-09-18T00:41:27.512984+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:08:28.112030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094839100000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3706,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:29.810182+0000","last_change":"2024-09-17T23:31:24.091376+0000","last_active":"2024-09-18T00:41:29.810182+0000","last_peered":"2024-09-18T00:41:29.810182+0000","last_clean":"2024-09-18T00:41:29.810182+0000","last_became_active":"2024-09-17T23:31:21.496953+0000","last_became_peered":"2024-09-17T23:31:21.496953+0000","last_unstale":"2024-09-18T00:41:29.810182+0000","last_undegraded":"2024-09-18T00:41:29.810182+0000","last_fullsized":"2024-09-18T00:41:29.810182+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:40:43.130208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031250800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3707,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:29.810182+0000","last_change":"2024-09-17T23:31:24.091323+0000","last_active":"2024-09-18T00:41:29.810182+0000","last_peered":"2024-09-18T00:41:29.810182+0000","last_clean":"2024-09-18T00:41:29.810182+0000","last_became_active":"2024-09-17T23:31:21.497936+0000","last_became_peered":"2024-09-17T23:31:21.497936+0000","last_unstale":"2024-09-18T00:41:29.810182+0000","last_undegraded":"2024-09-18T00:41:29.810182+0000","last_fullsized":"2024-09-18T00:41:29.810182+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:22:43.988917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030225,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":3707,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.150484+0000","last_change":"2024-09-17T23:31:23.506776+0000","last_active":"2024-09-18T00:41:27.150484+0000","last_peered":"2024-09-18T00:41:27.150484+0000","last_clean":"2024-09-18T00:41:27.150484+0000","last_became_active":"2024-09-17T23:31:21.497054+0000","last_became_peered":"2024-09-17T23:31:21.497054+0000","last_unstale":"2024-09-18T00:41:27.150484+0000","last_undegraded":"2024-09-18T00:41:27.150484+0000","last_fullsized":"2024-09-18T00:41:27.150484+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:20.475375+0000","last_clean_scrub_stamp":"2024-09-17T23:31:20.475375+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:43:27.049283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051159800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"813'125","reported_seq":3871,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.149981+0000","last_change":"2024-09-17T23:31:15.637809+0000","last_active":"2024-09-18T00:41:27.149981+0000","last_peered":"2024-09-18T00:41:27.149981+0000","last_clean":"2024-09-18T00:41:27.149981+0000","last_became_active":"2024-09-17T23:31:15.636714+0000","last_became_peered":"2024-09-17T23:31:15.636714+0000","last_unstale":"2024-09-18T00:41:27.149981+0000","last_undegraded":"2024-09-18T00:41:27.149981+0000","last_fullsized":"2024-09-18T00:41:27.149981+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:31:14.466759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:31:14.466759+0000","last_clean_scrub_stamp":"2024-09-17T23:31:14.466759+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:05:04.207207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":113,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":113,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6090,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706513,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":816628,"kb_used_data":5216,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":93555212,"statfs":{"total":96636764160,"available":95800537088,"internally_reserved":0,"allocated":5341184,"data_stored":2964246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739216,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726448,"kb_used_data":5276,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93645392,"statfs":{"total":96636764160,"available":95892881408,"internally_reserved":0,"allocated":5402624,"data_stored":3554633,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739214,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":902752,"kb_used_data":5264,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":93469088,"statfs":{"total":96636764160,"available":95712346112,"internally_reserved":0,"allocated":5390336,"data_stored":3550943,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T00:41:34.152 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-17T23:31:20.475375+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=31, tm_sec=20, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=41, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:41:34.152 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-17T23:31:20.475375+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=31, tm_sec=20, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=41, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:41:34.152 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-17T23:31:20.475375+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=31, tm_sec=20, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=41, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:41:34.152 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-17T23:31:20.475375+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=31, tm_sec=20, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=41, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:41:34.152 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-17T23:31:20.475375+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=31, tm_sec=20, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=41, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:41:34.153 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-17T23:31:20.475375+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=31, tm_sec=20, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=41, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:41:34.153 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-17T23:31:20.475375+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=31, tm_sec=20, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=41, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:41:34.153 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-17T23:31:20.475375+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=31, tm_sec=20, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=41, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:41:34.153 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-17T23:31:14.466759+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=31, tm_sec=14, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=41, tm_sec=31, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:41:34.153 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T00:41:54.154 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-18T00:41:54.445 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-09-18T00:41:54.445 INFO:teuthology.orchestra.run.smithi192.stderr:dumped all 2024-09-18T00:41:54.457 INFO:teuthology.orchestra.run.smithi192.stdout:{"pg_ready":true,"pg_map":{"version":3959,"stamp":"2024-09-18T00:41:52.791672+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":130,"num_read_kb":113,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2445832,"kb_used_data":15768,"kb_used_omap":4,"kb_used_meta":2430011,"kb_avail":280669688,"statfs":{"total":289910292480,"available":287405760512,"internally_reserved":0,"allocated":16146432,"data_stored":10073512,"data_compressed":6090,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":2488331599},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002405"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3714,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:39.130278+0000","last_change":"2024-09-18T00:41:39.130299+0000","last_active":"2024-09-18T00:41:39.130278+0000","last_peered":"2024-09-18T00:41:39.130278+0000","last_clean":"2024-09-18T00:41:39.130278+0000","last_became_active":"2024-09-17T23:31:21.498864+0000","last_became_peered":"2024-09-17T23:31:21.498864+0000","last_unstale":"2024-09-18T00:41:39.130278+0000","last_undegraded":"2024-09-18T00:41:39.130278+0000","last_fullsized":"2024-09-18T00:41:39.130278+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T00:41:39.130178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T00:41:39.130178+0000","last_clean_scrub_stamp":"2024-09-18T00:41:39.130178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:11:47.785365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083165799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3714,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:38.161408+0000","last_change":"2024-09-18T00:41:38.161452+0000","last_active":"2024-09-18T00:41:38.161408+0000","last_peered":"2024-09-18T00:41:38.161408+0000","last_clean":"2024-09-18T00:41:38.161408+0000","last_became_active":"2024-09-17T23:31:21.498384+0000","last_became_peered":"2024-09-17T23:31:21.498384+0000","last_unstale":"2024-09-18T00:41:38.161408+0000","last_undegraded":"2024-09-18T00:41:38.161408+0000","last_fullsized":"2024-09-18T00:41:38.161408+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T00:41:38.161298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T00:41:38.161298+0000","last_clean_scrub_stamp":"2024-09-18T00:41:38.161298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:11:14.193496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010814310000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3715,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:37.154319+0000","last_change":"2024-09-18T00:41:37.154357+0000","last_active":"2024-09-18T00:41:37.154319+0000","last_peered":"2024-09-18T00:41:37.154319+0000","last_clean":"2024-09-18T00:41:37.154319+0000","last_became_active":"2024-09-17T23:31:21.497851+0000","last_became_peered":"2024-09-17T23:31:21.497851+0000","last_unstale":"2024-09-18T00:41:37.154319+0000","last_undegraded":"2024-09-18T00:41:37.154319+0000","last_fullsized":"2024-09-18T00:41:37.154319+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T00:41:37.154196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T00:41:37.154196+0000","last_clean_scrub_stamp":"2024-09-18T00:41:37.154196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:13:02.114118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00049346500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3715,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:36.187207+0000","last_change":"2024-09-18T00:41:36.187230+0000","last_active":"2024-09-18T00:41:36.187207+0000","last_peered":"2024-09-18T00:41:36.187207+0000","last_clean":"2024-09-18T00:41:36.187207+0000","last_became_active":"2024-09-17T23:31:21.497623+0000","last_became_peered":"2024-09-17T23:31:21.497623+0000","last_unstale":"2024-09-18T00:41:36.187207+0000","last_undegraded":"2024-09-18T00:41:36.187207+0000","last_fullsized":"2024-09-18T00:41:36.187207+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T00:41:36.187114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T00:41:36.187114+0000","last_clean_scrub_stamp":"2024-09-18T00:41:36.187114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:13:21.341213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079303000000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"1842'3","reported_seq":3717,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:33.480471+0000","last_change":"2024-09-18T00:41:33.480495+0000","last_active":"2024-09-18T00:41:33.480471+0000","last_peered":"2024-09-18T00:41:33.480471+0000","last_clean":"2024-09-18T00:41:33.480471+0000","last_became_active":"2024-09-17T23:31:21.498502+0000","last_became_peered":"2024-09-17T23:31:21.498502+0000","last_unstale":"2024-09-18T00:41:33.480471+0000","last_undegraded":"2024-09-18T00:41:33.480471+0000","last_fullsized":"2024-09-18T00:41:33.480471+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"1842'3","last_scrub_stamp":"2024-09-18T00:41:33.480401+0000","last_deep_scrub":"1842'3","last_deep_scrub_stamp":"2024-09-18T00:41:33.480401+0000","last_clean_scrub_stamp":"2024-09-18T00:41:33.480401+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:02:26.850586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00094839100000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3715,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:34.785811+0000","last_change":"2024-09-18T00:41:34.785832+0000","last_active":"2024-09-18T00:41:34.785811+0000","last_peered":"2024-09-18T00:41:34.785811+0000","last_clean":"2024-09-18T00:41:34.785811+0000","last_became_active":"2024-09-17T23:31:21.496953+0000","last_became_peered":"2024-09-17T23:31:21.496953+0000","last_unstale":"2024-09-18T00:41:34.785811+0000","last_undegraded":"2024-09-18T00:41:34.785811+0000","last_fullsized":"2024-09-18T00:41:34.785811+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T00:41:34.785722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T00:41:34.785722+0000","last_clean_scrub_stamp":"2024-09-18T00:41:34.785722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:40:04.133547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031250800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3717,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:35.805207+0000","last_change":"2024-09-18T00:41:35.805231+0000","last_active":"2024-09-18T00:41:35.805207+0000","last_peered":"2024-09-18T00:41:35.805207+0000","last_clean":"2024-09-18T00:41:35.805207+0000","last_became_active":"2024-09-17T23:31:21.497936+0000","last_became_peered":"2024-09-17T23:31:21.497936+0000","last_unstale":"2024-09-18T00:41:35.805207+0000","last_undegraded":"2024-09-18T00:41:35.805207+0000","last_fullsized":"2024-09-18T00:41:35.805207+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T00:41:35.805113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T00:41:35.805113+0000","last_clean_scrub_stamp":"2024-09-18T00:41:35.805113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:27:42.736112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030225,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"1842'2","reported_seq":3717,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:35.157703+0000","last_change":"2024-09-18T00:41:35.157729+0000","last_active":"2024-09-18T00:41:35.157703+0000","last_peered":"2024-09-18T00:41:35.157703+0000","last_clean":"2024-09-18T00:41:35.157703+0000","last_became_active":"2024-09-17T23:31:21.497054+0000","last_became_peered":"2024-09-17T23:31:21.497054+0000","last_unstale":"2024-09-18T00:41:35.157703+0000","last_undegraded":"2024-09-18T00:41:35.157703+0000","last_fullsized":"2024-09-18T00:41:35.157703+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"1842'2","last_scrub_stamp":"2024-09-18T00:41:35.157615+0000","last_deep_scrub":"1842'2","last_deep_scrub_stamp":"2024-09-18T00:41:35.157615+0000","last_clean_scrub_stamp":"2024-09-18T00:41:35.157615+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:41:22.762817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051159800000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"1842'127","reported_seq":3882,"reported_epoch":1842,"state":"active+clean","last_fresh":"2024-09-18T00:41:34.162888+0000","last_change":"2024-09-18T00:41:34.162939+0000","last_active":"2024-09-18T00:41:34.162888+0000","last_peered":"2024-09-18T00:41:34.162888+0000","last_clean":"2024-09-18T00:41:34.162888+0000","last_became_active":"2024-09-17T23:31:15.636714+0000","last_became_peered":"2024-09-17T23:31:15.636714+0000","last_unstale":"2024-09-18T00:41:34.162888+0000","last_undegraded":"2024-09-18T00:41:34.162888+0000","last_fullsized":"2024-09-18T00:41:34.162888+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"1842'127","last_scrub_stamp":"2024-09-18T00:41:34.162803+0000","last_deep_scrub":"1842'127","last_deep_scrub_stamp":"2024-09-18T00:41:34.162803+0000","last_clean_scrub_stamp":"2024-09-18T00:41:34.162803+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:47:22.965124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":113,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":113,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6090,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654706517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":816624,"kb_used_data":5216,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":93555216,"statfs":{"total":96636764160,"available":95800541184,"internally_reserved":0,"allocated":5341184,"data_stored":2964246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":830863813},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739220,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726444,"kb_used_data":5276,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93645396,"statfs":{"total":96636764160,"available":95892885504,"internally_reserved":0,"allocated":5402624,"data_stored":3554633,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":738458053},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739218,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":902764,"kb_used_data":5276,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":93469076,"statfs":{"total":96636764160,"available":95712333824,"internally_reserved":0,"allocated":5402624,"data_stored":3554633,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3045,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T00:41:54.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-18T00:41:54.752 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-18T00:41:54.752 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-18T00:41:54.752 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-18T00:41:54.752 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T00:41:56.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-09-18T00:42:00.855 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-18T00:42:00.855 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-18T00:42:00.855 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T00:42:00.952 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-18T00:42:00.952 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-18T00:42:00.953 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T00:42:01.052 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-18T00:42:01.052 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-18T00:42:01.052 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-18T00:42:01.052 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T00:42:01.082 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-18T00:42:01.082 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-18T00:42:01.082 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-18T00:42:01.083 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T00:42:01.102 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-18T00:42:01.102 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-18T00:42:01.102 DEBUG:teuthology.orchestra.run.smithi192:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-18T00:42:01.321 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi192.front.sepia.ceph.com 2024-09-18T00:42:01.321 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-18T00:42:01.575 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi192.front.sepia.ceph.com 2024-09-18T00:42:01.575 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-18T00:42:01.654 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi192.front.sepia.ceph.com 2024-09-18T00:42:01.654 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-18T00:42:01.713 INFO:tasks.ceph:Archiving mon data... 2024-09-18T00:42:01.715 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909591/data/cluster2.mon.a.tgz 2024-09-18T00:42:01.715 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-09-18T00:42:01.730 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.5amgoQrOiQ 2024-09-18T00:42:01.730 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.5amgoQrOiQ 2024-09-18T00:42:01.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 has been restored 2024-09-18T00:42:01.883 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.5amgoQrOiQ 2024-09-18T00:42:01.967 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.5amgoQrOiQ is 503KB 2024-09-18T00:42:02.067 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.5amgoQrOiQ 2024-09-18T00:42:02.082 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-18T00:42:02.082 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T00:42:02.085 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T00:42:02.284 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-18T00:42:02.294 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-18T00:42:02.294 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-18T00:42:02.702 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-18T00:42:02.703 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-18T00:42:02.714 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":4022,"stamp":"2024-09-18T00:42:01.407715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3638812,"kb_used_data":17568,"kb_used_omap":4,"kb_used_meta":3621179,"kb_avail":279476708,"statfs":{"total":289910292480,"available":286184148992,"internally_reserved":0,"allocated":17989632,"data_stored":10999672,"data_compressed":6086,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":3708087631},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002724"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3781,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.064502+0000","last_change":"2024-09-17T23:30:43.479739+0000","last_active":"2024-09-18T00:41:26.064502+0000","last_peered":"2024-09-18T00:41:26.064502+0000","last_clean":"2024-09-18T00:41:26.064502+0000","last_became_active":"2024-09-17T23:30:41.472305+0000","last_became_peered":"2024-09-17T23:30:41.472305+0000","last_unstale":"2024-09-18T00:41:26.064502+0000","last_undegraded":"2024-09-18T00:41:26.064502+0000","last_fullsized":"2024-09-18T00:41:26.064502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:14:19.225045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050569899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3781,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.062938+0000","last_change":"2024-09-17T23:30:43.480063+0000","last_active":"2024-09-18T00:41:26.062938+0000","last_peered":"2024-09-18T00:41:26.062938+0000","last_clean":"2024-09-18T00:41:26.062938+0000","last_became_active":"2024-09-17T23:30:41.471899+0000","last_became_peered":"2024-09-17T23:30:41.471899+0000","last_unstale":"2024-09-18T00:41:26.062938+0000","last_undegraded":"2024-09-18T00:41:26.062938+0000","last_fullsized":"2024-09-18T00:41:26.062938+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:25:46.122761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082020499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3783,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.061909+0000","last_change":"2024-09-17T23:30:43.480349+0000","last_active":"2024-09-18T00:41:26.061909+0000","last_peered":"2024-09-18T00:41:26.061909+0000","last_clean":"2024-09-18T00:41:26.061909+0000","last_became_active":"2024-09-17T23:30:41.471801+0000","last_became_peered":"2024-09-17T23:30:41.471801+0000","last_unstale":"2024-09-18T00:41:26.061909+0000","last_undegraded":"2024-09-18T00:41:26.061909+0000","last_fullsized":"2024-09-18T00:41:26.061909+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:17:10.596752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00106452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3783,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.062901+0000","last_change":"2024-09-17T23:30:43.480110+0000","last_active":"2024-09-18T00:41:26.062901+0000","last_peered":"2024-09-18T00:41:26.062901+0000","last_clean":"2024-09-18T00:41:26.062901+0000","last_became_active":"2024-09-17T23:30:41.471678+0000","last_became_peered":"2024-09-17T23:30:41.471678+0000","last_unstale":"2024-09-18T00:41:26.062901+0000","last_undegraded":"2024-09-18T00:41:26.062901+0000","last_fullsized":"2024-09-18T00:41:26.062901+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:16:48.723032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088593199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":3785,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:28.287042+0000","last_change":"2024-09-17T23:30:43.476525+0000","last_active":"2024-09-18T00:41:28.287042+0000","last_peered":"2024-09-18T00:41:28.287042+0000","last_clean":"2024-09-18T00:41:28.287042+0000","last_became_active":"2024-09-17T23:30:41.470241+0000","last_became_peered":"2024-09-17T23:30:41.470241+0000","last_unstale":"2024-09-18T00:41:28.287042+0000","last_undegraded":"2024-09-18T00:41:28.287042+0000","last_fullsized":"2024-09-18T00:41:28.287042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:48:45.144310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051423599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3783,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.316878+0000","last_change":"2024-09-17T23:30:43.635997+0000","last_active":"2024-09-18T00:41:27.316878+0000","last_peered":"2024-09-18T00:41:27.316878+0000","last_clean":"2024-09-18T00:41:27.316878+0000","last_became_active":"2024-09-17T23:30:41.474920+0000","last_became_peered":"2024-09-17T23:30:41.474920+0000","last_unstale":"2024-09-18T00:41:27.316878+0000","last_undegraded":"2024-09-18T00:41:27.316878+0000","last_fullsized":"2024-09-18T00:41:27.316878+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:19:55.884592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041385499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3784,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.319352+0000","last_change":"2024-09-17T23:30:43.636000+0000","last_active":"2024-09-18T00:41:27.319352+0000","last_peered":"2024-09-18T00:41:27.319352+0000","last_clean":"2024-09-18T00:41:27.319352+0000","last_became_active":"2024-09-17T23:30:41.474871+0000","last_became_peered":"2024-09-17T23:30:41.474871+0000","last_unstale":"2024-09-18T00:41:27.319352+0000","last_undegraded":"2024-09-18T00:41:27.319352+0000","last_fullsized":"2024-09-18T00:41:27.319352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:06:22.909014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041168599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":3784,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.064070+0000","last_change":"2024-09-17T23:30:43.479732+0000","last_active":"2024-09-18T00:41:26.064070+0000","last_peered":"2024-09-18T00:41:26.064070+0000","last_clean":"2024-09-18T00:41:26.064070+0000","last_became_active":"2024-09-17T23:30:41.473769+0000","last_became_peered":"2024-09-17T23:30:41.473769+0000","last_unstale":"2024-09-18T00:41:26.064070+0000","last_undegraded":"2024-09-18T00:41:26.064070+0000","last_fullsized":"2024-09-18T00:41:26.064070+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:09:57.242769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054243800000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"794'125","reported_seq":3945,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.063531+0000","last_change":"2024-09-17T23:30:37.576035+0000","last_active":"2024-09-18T00:41:26.063531+0000","last_peered":"2024-09-18T00:41:26.063531+0000","last_clean":"2024-09-18T00:41:26.063531+0000","last_became_active":"2024-09-17T23:30:37.575069+0000","last_became_peered":"2024-09-17T23:30:37.575069+0000","last_unstale":"2024-09-18T00:41:26.063531+0000","last_undegraded":"2024-09-18T00:41:26.063531+0000","last_fullsized":"2024-09-18T00:41:26.063531+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:36.447566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:36.447566+0000","last_clean_scrub_stamp":"2024-09-17T23:30:36.447566+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:27:48.840294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6086,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739232,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1079632,"kb_used_data":5816,"kb_used_omap":1,"kb_used_meta":1073790,"kb_avail":93292208,"statfs":{"total":96636764160,"available":95531220992,"internally_reserved":0,"allocated":5955584,"data_stored":3272966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1099561413},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739232,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1196296,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":1190398,"kb_avail":93175544,"statfs":{"total":96636764160,"available":95411757056,"internally_reserved":0,"allocated":6017024,"data_stored":3863353,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1218968005},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739230,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1362884,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":1356990,"kb_avail":93008956,"statfs":{"total":96636764160,"available":95241170944,"internally_reserved":0,"allocated":6017024,"data_stored":3863353,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1389558213},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T00:42:02.715 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-18T00:42:02.990 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-18T00:42:02.990 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-18T00:42:03.002 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":4022,"stamp":"2024-09-18T00:42:01.407715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3638812,"kb_used_data":17568,"kb_used_omap":4,"kb_used_meta":3621179,"kb_avail":279476708,"statfs":{"total":289910292480,"available":286184148992,"internally_reserved":0,"allocated":17989632,"data_stored":10999672,"data_compressed":6086,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":3708087631},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002724"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3781,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.064502+0000","last_change":"2024-09-17T23:30:43.479739+0000","last_active":"2024-09-18T00:41:26.064502+0000","last_peered":"2024-09-18T00:41:26.064502+0000","last_clean":"2024-09-18T00:41:26.064502+0000","last_became_active":"2024-09-17T23:30:41.472305+0000","last_became_peered":"2024-09-17T23:30:41.472305+0000","last_unstale":"2024-09-18T00:41:26.064502+0000","last_undegraded":"2024-09-18T00:41:26.064502+0000","last_fullsized":"2024-09-18T00:41:26.064502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:14:19.225045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050569899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3781,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.062938+0000","last_change":"2024-09-17T23:30:43.480063+0000","last_active":"2024-09-18T00:41:26.062938+0000","last_peered":"2024-09-18T00:41:26.062938+0000","last_clean":"2024-09-18T00:41:26.062938+0000","last_became_active":"2024-09-17T23:30:41.471899+0000","last_became_peered":"2024-09-17T23:30:41.471899+0000","last_unstale":"2024-09-18T00:41:26.062938+0000","last_undegraded":"2024-09-18T00:41:26.062938+0000","last_fullsized":"2024-09-18T00:41:26.062938+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:25:46.122761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082020499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3783,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.061909+0000","last_change":"2024-09-17T23:30:43.480349+0000","last_active":"2024-09-18T00:41:26.061909+0000","last_peered":"2024-09-18T00:41:26.061909+0000","last_clean":"2024-09-18T00:41:26.061909+0000","last_became_active":"2024-09-17T23:30:41.471801+0000","last_became_peered":"2024-09-17T23:30:41.471801+0000","last_unstale":"2024-09-18T00:41:26.061909+0000","last_undegraded":"2024-09-18T00:41:26.061909+0000","last_fullsized":"2024-09-18T00:41:26.061909+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:17:10.596752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00106452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3783,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.062901+0000","last_change":"2024-09-17T23:30:43.480110+0000","last_active":"2024-09-18T00:41:26.062901+0000","last_peered":"2024-09-18T00:41:26.062901+0000","last_clean":"2024-09-18T00:41:26.062901+0000","last_became_active":"2024-09-17T23:30:41.471678+0000","last_became_peered":"2024-09-17T23:30:41.471678+0000","last_unstale":"2024-09-18T00:41:26.062901+0000","last_undegraded":"2024-09-18T00:41:26.062901+0000","last_fullsized":"2024-09-18T00:41:26.062901+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:16:48.723032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088593199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":3785,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:28.287042+0000","last_change":"2024-09-17T23:30:43.476525+0000","last_active":"2024-09-18T00:41:28.287042+0000","last_peered":"2024-09-18T00:41:28.287042+0000","last_clean":"2024-09-18T00:41:28.287042+0000","last_became_active":"2024-09-17T23:30:41.470241+0000","last_became_peered":"2024-09-17T23:30:41.470241+0000","last_unstale":"2024-09-18T00:41:28.287042+0000","last_undegraded":"2024-09-18T00:41:28.287042+0000","last_fullsized":"2024-09-18T00:41:28.287042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:48:45.144310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051423599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3783,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.316878+0000","last_change":"2024-09-17T23:30:43.635997+0000","last_active":"2024-09-18T00:41:27.316878+0000","last_peered":"2024-09-18T00:41:27.316878+0000","last_clean":"2024-09-18T00:41:27.316878+0000","last_became_active":"2024-09-17T23:30:41.474920+0000","last_became_peered":"2024-09-17T23:30:41.474920+0000","last_unstale":"2024-09-18T00:41:27.316878+0000","last_undegraded":"2024-09-18T00:41:27.316878+0000","last_fullsized":"2024-09-18T00:41:27.316878+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:19:55.884592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041385499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3784,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.319352+0000","last_change":"2024-09-17T23:30:43.636000+0000","last_active":"2024-09-18T00:41:27.319352+0000","last_peered":"2024-09-18T00:41:27.319352+0000","last_clean":"2024-09-18T00:41:27.319352+0000","last_became_active":"2024-09-17T23:30:41.474871+0000","last_became_peered":"2024-09-17T23:30:41.474871+0000","last_unstale":"2024-09-18T00:41:27.319352+0000","last_undegraded":"2024-09-18T00:41:27.319352+0000","last_fullsized":"2024-09-18T00:41:27.319352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:06:22.909014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041168599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":3784,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.064070+0000","last_change":"2024-09-17T23:30:43.479732+0000","last_active":"2024-09-18T00:41:26.064070+0000","last_peered":"2024-09-18T00:41:26.064070+0000","last_clean":"2024-09-18T00:41:26.064070+0000","last_became_active":"2024-09-17T23:30:41.473769+0000","last_became_peered":"2024-09-17T23:30:41.473769+0000","last_unstale":"2024-09-18T00:41:26.064070+0000","last_undegraded":"2024-09-18T00:41:26.064070+0000","last_fullsized":"2024-09-18T00:41:26.064070+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:09:57.242769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054243800000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"794'125","reported_seq":3945,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.063531+0000","last_change":"2024-09-17T23:30:37.576035+0000","last_active":"2024-09-18T00:41:26.063531+0000","last_peered":"2024-09-18T00:41:26.063531+0000","last_clean":"2024-09-18T00:41:26.063531+0000","last_became_active":"2024-09-17T23:30:37.575069+0000","last_became_peered":"2024-09-17T23:30:37.575069+0000","last_unstale":"2024-09-18T00:41:26.063531+0000","last_undegraded":"2024-09-18T00:41:26.063531+0000","last_fullsized":"2024-09-18T00:41:26.063531+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:36.447566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:36.447566+0000","last_clean_scrub_stamp":"2024-09-17T23:30:36.447566+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:27:48.840294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6086,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739232,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1079632,"kb_used_data":5816,"kb_used_omap":1,"kb_used_meta":1073790,"kb_avail":93292208,"statfs":{"total":96636764160,"available":95531220992,"internally_reserved":0,"allocated":5955584,"data_stored":3272966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1099561413},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739232,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1196296,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":1190398,"kb_avail":93175544,"statfs":{"total":96636764160,"available":95411757056,"internally_reserved":0,"allocated":6017024,"data_stored":3863353,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1218968005},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739230,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1362884,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":1356990,"kb_avail":93008956,"statfs":{"total":96636764160,"available":95241170944,"internally_reserved":0,"allocated":6017024,"data_stored":3863353,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1389558213},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T00:42:03.003 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-18T00:42:03.003 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-18T00:42:03.275 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-18T00:42:03.275 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-18T00:42:03.287 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":4022,"stamp":"2024-09-18T00:42:01.407715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3638812,"kb_used_data":17568,"kb_used_omap":4,"kb_used_meta":3621179,"kb_avail":279476708,"statfs":{"total":289910292480,"available":286184148992,"internally_reserved":0,"allocated":17989632,"data_stored":10999672,"data_compressed":6086,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":3708087631},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002724"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3781,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.064502+0000","last_change":"2024-09-17T23:30:43.479739+0000","last_active":"2024-09-18T00:41:26.064502+0000","last_peered":"2024-09-18T00:41:26.064502+0000","last_clean":"2024-09-18T00:41:26.064502+0000","last_became_active":"2024-09-17T23:30:41.472305+0000","last_became_peered":"2024-09-17T23:30:41.472305+0000","last_unstale":"2024-09-18T00:41:26.064502+0000","last_undegraded":"2024-09-18T00:41:26.064502+0000","last_fullsized":"2024-09-18T00:41:26.064502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:14:19.225045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050569899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3781,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.062938+0000","last_change":"2024-09-17T23:30:43.480063+0000","last_active":"2024-09-18T00:41:26.062938+0000","last_peered":"2024-09-18T00:41:26.062938+0000","last_clean":"2024-09-18T00:41:26.062938+0000","last_became_active":"2024-09-17T23:30:41.471899+0000","last_became_peered":"2024-09-17T23:30:41.471899+0000","last_unstale":"2024-09-18T00:41:26.062938+0000","last_undegraded":"2024-09-18T00:41:26.062938+0000","last_fullsized":"2024-09-18T00:41:26.062938+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:25:46.122761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082020499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3783,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.061909+0000","last_change":"2024-09-17T23:30:43.480349+0000","last_active":"2024-09-18T00:41:26.061909+0000","last_peered":"2024-09-18T00:41:26.061909+0000","last_clean":"2024-09-18T00:41:26.061909+0000","last_became_active":"2024-09-17T23:30:41.471801+0000","last_became_peered":"2024-09-17T23:30:41.471801+0000","last_unstale":"2024-09-18T00:41:26.061909+0000","last_undegraded":"2024-09-18T00:41:26.061909+0000","last_fullsized":"2024-09-18T00:41:26.061909+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:17:10.596752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00106452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3783,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.062901+0000","last_change":"2024-09-17T23:30:43.480110+0000","last_active":"2024-09-18T00:41:26.062901+0000","last_peered":"2024-09-18T00:41:26.062901+0000","last_clean":"2024-09-18T00:41:26.062901+0000","last_became_active":"2024-09-17T23:30:41.471678+0000","last_became_peered":"2024-09-17T23:30:41.471678+0000","last_unstale":"2024-09-18T00:41:26.062901+0000","last_undegraded":"2024-09-18T00:41:26.062901+0000","last_fullsized":"2024-09-18T00:41:26.062901+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:16:48.723032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088593199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":3785,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:28.287042+0000","last_change":"2024-09-17T23:30:43.476525+0000","last_active":"2024-09-18T00:41:28.287042+0000","last_peered":"2024-09-18T00:41:28.287042+0000","last_clean":"2024-09-18T00:41:28.287042+0000","last_became_active":"2024-09-17T23:30:41.470241+0000","last_became_peered":"2024-09-17T23:30:41.470241+0000","last_unstale":"2024-09-18T00:41:28.287042+0000","last_undegraded":"2024-09-18T00:41:28.287042+0000","last_fullsized":"2024-09-18T00:41:28.287042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:48:45.144310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051423599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3783,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.316878+0000","last_change":"2024-09-17T23:30:43.635997+0000","last_active":"2024-09-18T00:41:27.316878+0000","last_peered":"2024-09-18T00:41:27.316878+0000","last_clean":"2024-09-18T00:41:27.316878+0000","last_became_active":"2024-09-17T23:30:41.474920+0000","last_became_peered":"2024-09-17T23:30:41.474920+0000","last_unstale":"2024-09-18T00:41:27.316878+0000","last_undegraded":"2024-09-18T00:41:27.316878+0000","last_fullsized":"2024-09-18T00:41:27.316878+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:19:55.884592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041385499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3784,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.319352+0000","last_change":"2024-09-17T23:30:43.636000+0000","last_active":"2024-09-18T00:41:27.319352+0000","last_peered":"2024-09-18T00:41:27.319352+0000","last_clean":"2024-09-18T00:41:27.319352+0000","last_became_active":"2024-09-17T23:30:41.474871+0000","last_became_peered":"2024-09-17T23:30:41.474871+0000","last_unstale":"2024-09-18T00:41:27.319352+0000","last_undegraded":"2024-09-18T00:41:27.319352+0000","last_fullsized":"2024-09-18T00:41:27.319352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:06:22.909014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041168599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":3784,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.064070+0000","last_change":"2024-09-17T23:30:43.479732+0000","last_active":"2024-09-18T00:41:26.064070+0000","last_peered":"2024-09-18T00:41:26.064070+0000","last_clean":"2024-09-18T00:41:26.064070+0000","last_became_active":"2024-09-17T23:30:41.473769+0000","last_became_peered":"2024-09-17T23:30:41.473769+0000","last_unstale":"2024-09-18T00:41:26.064070+0000","last_undegraded":"2024-09-18T00:41:26.064070+0000","last_fullsized":"2024-09-18T00:41:26.064070+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:09:57.242769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054243800000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"794'125","reported_seq":3945,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.063531+0000","last_change":"2024-09-17T23:30:37.576035+0000","last_active":"2024-09-18T00:41:26.063531+0000","last_peered":"2024-09-18T00:41:26.063531+0000","last_clean":"2024-09-18T00:41:26.063531+0000","last_became_active":"2024-09-17T23:30:37.575069+0000","last_became_peered":"2024-09-17T23:30:37.575069+0000","last_unstale":"2024-09-18T00:41:26.063531+0000","last_undegraded":"2024-09-18T00:41:26.063531+0000","last_fullsized":"2024-09-18T00:41:26.063531+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:36.447566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:36.447566+0000","last_clean_scrub_stamp":"2024-09-17T23:30:36.447566+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:27:48.840294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6086,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739232,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1079632,"kb_used_data":5816,"kb_used_omap":1,"kb_used_meta":1073790,"kb_avail":93292208,"statfs":{"total":96636764160,"available":95531220992,"internally_reserved":0,"allocated":5955584,"data_stored":3272966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1099561413},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739232,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1196296,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":1190398,"kb_avail":93175544,"statfs":{"total":96636764160,"available":95411757056,"internally_reserved":0,"allocated":6017024,"data_stored":3863353,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1218968005},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739230,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1362884,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":1356990,"kb_avail":93008956,"statfs":{"total":96636764160,"available":95241170944,"internally_reserved":0,"allocated":6017024,"data_stored":3863353,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1389558213},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T00:42:03.288 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-18T00:42:03.566 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-18T00:42:03.566 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":1881,"fsid":"eeb17cc0-06d7-4ce4-845b-f0ddd9bde342","created":"2024-09-17T23:30:24.196667+0000","modified":"2024-09-18T00:41:26.054836+0000","last_up_change":"2024-09-17T23:30:34.445001+0000","last_in_change":"2024-09-17T23:30:25.039838+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T23:30:36.425699+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-17T23:30:39.605981+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"19bbd5f7-1cb2-4cb0-ac9a-2f7c843445cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6801","nonce":3675985324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6803","nonce":3675985324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6807","nonce":3675985324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":3675985324},{"type":"v1","addr":"172.21.15.96:6805","nonce":3675985324}]},"public_addr":"172.21.15.96:6801/3675985324","cluster_addr":"172.21.15.96:6803/3675985324","heartbeat_back_addr":"172.21.15.96:6807/3675985324","heartbeat_front_addr":"172.21.15.96:6805/3675985324","state":["exists","up"]},{"osd":1,"uuid":"1ab00cad-2baf-4f29-88de-ca8a63a7ac41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6817","nonce":3450214342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6819","nonce":3450214342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6823","nonce":3450214342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3450214342},{"type":"v1","addr":"172.21.15.96:6821","nonce":3450214342}]},"public_addr":"172.21.15.96:6817/3450214342","cluster_addr":"172.21.15.96:6819/3450214342","heartbeat_back_addr":"172.21.15.96:6823/3450214342","heartbeat_front_addr":"172.21.15.96:6821/3450214342","state":["exists","up"]},{"osd":2,"uuid":"53a69aca-c21a-4d27-af52-f06da4af6119","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6809","nonce":2203324171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6811","nonce":2203324171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6815","nonce":2203324171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":2203324171},{"type":"v1","addr":"172.21.15.96:6813","nonce":2203324171}]},"public_addr":"172.21.15.96:6809/2203324171","cluster_addr":"172.21.15.96:6811/2203324171","heartbeat_back_addr":"172.21.15.96:6815/2203324171","heartbeat_front_addr":"172.21.15.96:6813/2203324171","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:30:31.673547+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:30:31.891181+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T23:30:31.700724+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":3281,"length":1},{"begin":3290,"length":1},{"begin":3296,"length":1},{"begin":3299,"length":1},{"begin":3304,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T00:42:04.580 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-18T00:42:04.581 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-18T00:42:04.707 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-18T00:42:04.707 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T00:42:04.708 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-18T00:42:04.718 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-18T00:42:04.994 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 0 to deep-scrub 2024-09-18T00:42:05.006 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-18T00:42:05.007 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-18T00:42:05.128 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-18T00:42:05.129 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T00:42:05.129 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-18T00:42:05.139 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-18T00:42:05.415 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 1 to deep-scrub 2024-09-18T00:42:05.428 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-18T00:42:05.428 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-18T00:42:05.553 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-18T00:42:05.553 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T00:42:05.554 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-18T00:42:05.564 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-18T00:42:05.845 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 2 to deep-scrub 2024-09-18T00:42:05.858 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-18T00:42:06.131 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-18T00:42:06.131 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-18T00:42:06.144 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":4024,"stamp":"2024-09-18T00:42:05.408393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3638812,"kb_used_data":17568,"kb_used_omap":4,"kb_used_meta":3621179,"kb_avail":279476708,"statfs":{"total":289910292480,"available":286184148992,"internally_reserved":0,"allocated":17989632,"data_stored":10999672,"data_compressed":6086,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":3708087631},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002443"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3781,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.064502+0000","last_change":"2024-09-17T23:30:43.479739+0000","last_active":"2024-09-18T00:41:26.064502+0000","last_peered":"2024-09-18T00:41:26.064502+0000","last_clean":"2024-09-18T00:41:26.064502+0000","last_became_active":"2024-09-17T23:30:41.472305+0000","last_became_peered":"2024-09-17T23:30:41.472305+0000","last_unstale":"2024-09-18T00:41:26.064502+0000","last_undegraded":"2024-09-18T00:41:26.064502+0000","last_fullsized":"2024-09-18T00:41:26.064502+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T00:14:19.225045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050569899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3781,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.062938+0000","last_change":"2024-09-17T23:30:43.480063+0000","last_active":"2024-09-18T00:41:26.062938+0000","last_peered":"2024-09-18T00:41:26.062938+0000","last_clean":"2024-09-18T00:41:26.062938+0000","last_became_active":"2024-09-17T23:30:41.471899+0000","last_became_peered":"2024-09-17T23:30:41.471899+0000","last_unstale":"2024-09-18T00:41:26.062938+0000","last_undegraded":"2024-09-18T00:41:26.062938+0000","last_fullsized":"2024-09-18T00:41:26.062938+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:25:46.122761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082020499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3783,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.061909+0000","last_change":"2024-09-17T23:30:43.480349+0000","last_active":"2024-09-18T00:41:26.061909+0000","last_peered":"2024-09-18T00:41:26.061909+0000","last_clean":"2024-09-18T00:41:26.061909+0000","last_became_active":"2024-09-17T23:30:41.471801+0000","last_became_peered":"2024-09-17T23:30:41.471801+0000","last_unstale":"2024-09-18T00:41:26.061909+0000","last_undegraded":"2024-09-18T00:41:26.061909+0000","last_fullsized":"2024-09-18T00:41:26.061909+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T09:17:10.596752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00106452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3783,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.062901+0000","last_change":"2024-09-17T23:30:43.480110+0000","last_active":"2024-09-18T00:41:26.062901+0000","last_peered":"2024-09-18T00:41:26.062901+0000","last_clean":"2024-09-18T00:41:26.062901+0000","last_became_active":"2024-09-17T23:30:41.471678+0000","last_became_peered":"2024-09-17T23:30:41.471678+0000","last_unstale":"2024-09-18T00:41:26.062901+0000","last_undegraded":"2024-09-18T00:41:26.062901+0000","last_fullsized":"2024-09-18T00:41:26.062901+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T03:16:48.723032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088593199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":3785,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:28.287042+0000","last_change":"2024-09-17T23:30:43.476525+0000","last_active":"2024-09-18T00:41:28.287042+0000","last_peered":"2024-09-18T00:41:28.287042+0000","last_clean":"2024-09-18T00:41:28.287042+0000","last_became_active":"2024-09-17T23:30:41.470241+0000","last_became_peered":"2024-09-17T23:30:41.470241+0000","last_unstale":"2024-09-18T00:41:28.287042+0000","last_undegraded":"2024-09-18T00:41:28.287042+0000","last_fullsized":"2024-09-18T00:41:28.287042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:48:45.144310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051423599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3783,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.316878+0000","last_change":"2024-09-17T23:30:43.635997+0000","last_active":"2024-09-18T00:41:27.316878+0000","last_peered":"2024-09-18T00:41:27.316878+0000","last_clean":"2024-09-18T00:41:27.316878+0000","last_became_active":"2024-09-17T23:30:41.474920+0000","last_became_peered":"2024-09-17T23:30:41.474920+0000","last_unstale":"2024-09-18T00:41:27.316878+0000","last_undegraded":"2024-09-18T00:41:27.316878+0000","last_fullsized":"2024-09-18T00:41:27.316878+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T02:19:55.884592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041385499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3784,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:27.319352+0000","last_change":"2024-09-17T23:30:43.636000+0000","last_active":"2024-09-18T00:41:27.319352+0000","last_peered":"2024-09-18T00:41:27.319352+0000","last_clean":"2024-09-18T00:41:27.319352+0000","last_became_active":"2024-09-17T23:30:41.474871+0000","last_became_peered":"2024-09-17T23:30:41.474871+0000","last_unstale":"2024-09-18T00:41:27.319352+0000","last_undegraded":"2024-09-18T00:41:27.319352+0000","last_fullsized":"2024-09-18T00:41:27.319352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T04:06:22.909014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041168599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":3784,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.064070+0000","last_change":"2024-09-17T23:30:43.479732+0000","last_active":"2024-09-18T00:41:26.064070+0000","last_peered":"2024-09-18T00:41:26.064070+0000","last_clean":"2024-09-18T00:41:26.064070+0000","last_became_active":"2024-09-17T23:30:41.473769+0000","last_became_peered":"2024-09-17T23:30:41.473769+0000","last_unstale":"2024-09-18T00:41:26.064070+0000","last_undegraded":"2024-09-18T00:41:26.064070+0000","last_fullsized":"2024-09-18T00:41:26.064070+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:40.457072+0000","last_clean_scrub_stamp":"2024-09-17T23:30:40.457072+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:09:57.242769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054243800000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"794'125","reported_seq":3945,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:41:26.063531+0000","last_change":"2024-09-17T23:30:37.576035+0000","last_active":"2024-09-18T00:41:26.063531+0000","last_peered":"2024-09-18T00:41:26.063531+0000","last_clean":"2024-09-18T00:41:26.063531+0000","last_became_active":"2024-09-17T23:30:37.575069+0000","last_became_peered":"2024-09-17T23:30:37.575069+0000","last_unstale":"2024-09-18T00:41:26.063531+0000","last_undegraded":"2024-09-18T00:41:26.063531+0000","last_fullsized":"2024-09-18T00:41:26.063531+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T23:30:36.447566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T23:30:36.447566+0000","last_clean_scrub_stamp":"2024-09-17T23:30:36.447566+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:27:48.840294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6086,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":125,"ondisk_log_size":125,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739232,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1079632,"kb_used_data":5816,"kb_used_omap":1,"kb_used_meta":1073790,"kb_avail":93292208,"statfs":{"total":96636764160,"available":95531220992,"internally_reserved":0,"allocated":5955584,"data_stored":3272966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1099561413},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739232,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1196296,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":1190398,"kb_avail":93175544,"statfs":{"total":96636764160,"available":95411757056,"internally_reserved":0,"allocated":6017024,"data_stored":3863353,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1218968005},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739230,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1362884,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":1356990,"kb_avail":93008956,"statfs":{"total":96636764160,"available":95241170944,"internally_reserved":0,"allocated":6017024,"data_stored":3863353,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1389558213},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T00:42:06.145 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-17T23:30:40.457072+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=30, tm_sec=40, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=42, tm_sec=3, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:42:06.146 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-17T23:30:40.457072+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=30, tm_sec=40, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=42, tm_sec=3, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:42:06.146 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-17T23:30:40.457072+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=30, tm_sec=40, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=42, tm_sec=3, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:42:06.146 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-17T23:30:40.457072+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=30, tm_sec=40, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=42, tm_sec=3, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:42:06.146 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-17T23:30:40.457072+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=30, tm_sec=40, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=42, tm_sec=3, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:42:06.146 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-17T23:30:40.457072+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=30, tm_sec=40, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=42, tm_sec=3, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:42:06.147 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-17T23:30:40.457072+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=30, tm_sec=40, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=42, tm_sec=3, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:42:06.147 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-17T23:30:40.457072+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=30, tm_sec=40, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=42, tm_sec=3, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:42:06.147 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-17T23:30:36.447566+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=17, tm_hour=23, tm_min=30, tm_sec=36, tm_wday=1, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=0, tm_min=42, tm_sec=3, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T00:42:06.147 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T00:42:26.148 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-18T00:42:26.434 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-18T00:42:26.434 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-18T00:42:26.446 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":4034,"stamp":"2024-09-18T00:42:25.412599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":171,"num_write_kb":2550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3638800,"kb_used_data":17568,"kb_used_omap":4,"kb_used_meta":3621179,"kb_avail":279476720,"statfs":{"total":289910292480,"available":286184161280,"internally_reserved":0,"allocated":17989632,"data_stored":10999672,"data_compressed":6086,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":3708087631},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002448"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":3791,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:42:08.720198+0000","last_change":"2024-09-18T00:42:08.720198+0000","last_active":"2024-09-18T00:42:08.720198+0000","last_peered":"2024-09-18T00:42:08.720198+0000","last_clean":"2024-09-18T00:42:08.720198+0000","last_became_active":"2024-09-17T23:30:41.472305+0000","last_became_peered":"2024-09-17T23:30:41.472305+0000","last_unstale":"2024-09-18T00:42:08.720198+0000","last_undegraded":"2024-09-18T00:42:08.720198+0000","last_fullsized":"2024-09-18T00:42:08.720198+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T00:42:08.720060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T00:42:08.720060+0000","last_clean_scrub_stamp":"2024-09-18T00:42:08.720060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T07:11:56.625380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050569899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":3791,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:42:09.738674+0000","last_change":"2024-09-18T00:42:09.738674+0000","last_active":"2024-09-18T00:42:09.738674+0000","last_peered":"2024-09-18T00:42:09.738674+0000","last_clean":"2024-09-18T00:42:09.738674+0000","last_became_active":"2024-09-17T23:30:41.471899+0000","last_became_peered":"2024-09-17T23:30:41.471899+0000","last_unstale":"2024-09-18T00:42:09.738674+0000","last_undegraded":"2024-09-18T00:42:09.738674+0000","last_fullsized":"2024-09-18T00:42:09.738674+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T00:42:09.738534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T00:42:09.738534+0000","last_clean_scrub_stamp":"2024-09-18T00:42:09.738534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:41:52.023913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082020499999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":3792,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:42:07.754810+0000","last_change":"2024-09-18T00:42:07.754835+0000","last_active":"2024-09-18T00:42:07.754810+0000","last_peered":"2024-09-18T00:42:07.754810+0000","last_clean":"2024-09-18T00:42:07.754810+0000","last_became_active":"2024-09-17T23:30:41.471801+0000","last_became_peered":"2024-09-17T23:30:41.471801+0000","last_unstale":"2024-09-18T00:42:07.754810+0000","last_undegraded":"2024-09-18T00:42:07.754810+0000","last_fullsized":"2024-09-18T00:42:07.754810+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T00:42:07.754715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T00:42:07.754715+0000","last_clean_scrub_stamp":"2024-09-18T00:42:07.754715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T01:21:52.190405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00106452,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3792,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:42:06.705578+0000","last_change":"2024-09-18T00:42:06.705597+0000","last_active":"2024-09-18T00:42:06.705578+0000","last_peered":"2024-09-18T00:42:06.705578+0000","last_clean":"2024-09-18T00:42:06.705578+0000","last_became_active":"2024-09-17T23:30:41.471678+0000","last_became_peered":"2024-09-17T23:30:41.471678+0000","last_unstale":"2024-09-18T00:42:06.705578+0000","last_undegraded":"2024-09-18T00:42:06.705578+0000","last_fullsized":"2024-09-18T00:42:06.705578+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T00:42:06.705487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T00:42:06.705487+0000","last_clean_scrub_stamp":"2024-09-18T00:42:06.705487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T05:47:35.984692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088593199999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"1881'3","reported_seq":3795,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:42:05.587129+0000","last_change":"2024-09-18T00:42:05.587158+0000","last_active":"2024-09-18T00:42:05.587129+0000","last_peered":"2024-09-18T00:42:05.587129+0000","last_clean":"2024-09-18T00:42:05.587129+0000","last_became_active":"2024-09-17T23:30:41.470241+0000","last_became_peered":"2024-09-17T23:30:41.470241+0000","last_unstale":"2024-09-18T00:42:05.587129+0000","last_undegraded":"2024-09-18T00:42:05.587129+0000","last_fullsized":"2024-09-18T00:42:05.587129+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"1881'3","last_scrub_stamp":"2024-09-18T00:42:05.587057+0000","last_deep_scrub":"1881'3","last_deep_scrub_stamp":"2024-09-18T00:42:05.587057+0000","last_clean_scrub_stamp":"2024-09-18T00:42:05.587057+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T12:09:47.239573+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00051423599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":3792,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:42:07.842805+0000","last_change":"2024-09-18T00:42:07.842838+0000","last_active":"2024-09-18T00:42:07.842805+0000","last_peered":"2024-09-18T00:42:07.842805+0000","last_clean":"2024-09-18T00:42:07.842805+0000","last_became_active":"2024-09-17T23:30:41.474920+0000","last_became_peered":"2024-09-17T23:30:41.474920+0000","last_unstale":"2024-09-18T00:42:07.842805+0000","last_undegraded":"2024-09-18T00:42:07.842805+0000","last_fullsized":"2024-09-18T00:42:07.842805+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T00:42:07.842714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T00:42:07.842714+0000","last_clean_scrub_stamp":"2024-09-18T00:42:07.842714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T08:09:18.330952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041385499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":3793,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:42:06.806410+0000","last_change":"2024-09-18T00:42:06.806431+0000","last_active":"2024-09-18T00:42:06.806410+0000","last_peered":"2024-09-18T00:42:06.806410+0000","last_clean":"2024-09-18T00:42:06.806410+0000","last_became_active":"2024-09-17T23:30:41.474871+0000","last_became_peered":"2024-09-17T23:30:41.474871+0000","last_unstale":"2024-09-18T00:42:06.806410+0000","last_undegraded":"2024-09-18T00:42:06.806410+0000","last_fullsized":"2024-09-18T00:42:06.806410+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T00:42:06.806328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T00:42:06.806328+0000","last_clean_scrub_stamp":"2024-09-18T00:42:06.806328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:37:12.205100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041168599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"1881'2","reported_seq":3794,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:42:10.760142+0000","last_change":"2024-09-18T00:42:10.760204+0000","last_active":"2024-09-18T00:42:10.760142+0000","last_peered":"2024-09-18T00:42:10.760142+0000","last_clean":"2024-09-18T00:42:10.760142+0000","last_became_active":"2024-09-17T23:30:41.473769+0000","last_became_peered":"2024-09-17T23:30:41.473769+0000","last_unstale":"2024-09-18T00:42:10.760142+0000","last_undegraded":"2024-09-18T00:42:10.760142+0000","last_fullsized":"2024-09-18T00:42:10.760142+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"1881'2","last_scrub_stamp":"2024-09-18T00:42:10.760037+0000","last_deep_scrub":"1881'2","last_deep_scrub_stamp":"2024-09-18T00:42:10.760037+0000","last_clean_scrub_stamp":"2024-09-18T00:42:10.760037+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:12:40.383650+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00054243800000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"1881'127","reported_seq":3956,"reported_epoch":1881,"state":"active+clean","last_fresh":"2024-09-18T00:42:05.715378+0000","last_change":"2024-09-18T00:42:05.715434+0000","last_active":"2024-09-18T00:42:05.715378+0000","last_peered":"2024-09-18T00:42:05.715378+0000","last_clean":"2024-09-18T00:42:05.715378+0000","last_became_active":"2024-09-17T23:30:37.575069+0000","last_became_peered":"2024-09-17T23:30:37.575069+0000","last_unstale":"2024-09-18T00:42:05.715378+0000","last_undegraded":"2024-09-18T00:42:05.715378+0000","last_fullsized":"2024-09-18T00:42:05.715378+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"1881'127","last_scrub_stamp":"2024-09-18T00:42:05.715295+0000","last_deep_scrub":"1881'127","last_deep_scrub_stamp":"2024-09-18T00:42:05.715295+0000","last_clean_scrub_stamp":"2024-09-18T00:42:05.715295+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T06:25:54.944248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":169,"num_write_kb":2548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":6086,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739236,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1079624,"kb_used_data":5816,"kb_used_omap":1,"kb_used_meta":1073790,"kb_avail":93292216,"statfs":{"total":96636764160,"available":95531229184,"internally_reserved":0,"allocated":5955584,"data_stored":3272966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1099561413},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359739236,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1196292,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":1190398,"kb_avail":93175548,"statfs":{"total":96636764160,"available":95411761152,"internally_reserved":0,"allocated":6017024,"data_stored":3863353,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1218968005},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359739234,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1362884,"kb_used_data":5876,"kb_used_omap":1,"kb_used_meta":1356990,"kb_avail":93008956,"statfs":{"total":96636764160,"available":95241170944,"internally_reserved":0,"allocated":6017024,"data_stored":3863353,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1389558213},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":3043,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T00:42:26.447 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-18T00:42:26.737 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-18T00:42:26.737 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-18T00:42:26.737 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-18T00:42:26.737 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T00:42:26.838 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-18T00:42:26.838 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-18T00:42:26.838 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T00:42:26.934 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-18T00:42:26.934 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-18T00:42:26.935 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T00:42:27.022 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-18T00:42:27.022 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-18T00:42:27.022 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-18T00:42:27.022 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T00:42:27.048 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-18T00:42:27.048 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-18T00:42:27.048 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-18T00:42:27.049 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T00:42:27.104 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-18T00:42:27.104 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-18T00:42:27.105 DEBUG:teuthology.orchestra.run.smithi096:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-18T00:42:27.201 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-18T00:42:27.202 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-18T00:42:27.472 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-18T00:42:27.472 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-18T00:42:27.608 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-18T00:42:27.608 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-18T00:42:27.710 INFO:tasks.ceph:Archiving mon data... 2024-09-18T00:42:27.710 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909591/data/cluster1.mon.a.tgz 2024-09-18T00:42:27.710 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-09-18T00:42:27.726 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.Fa4umRLnwz 2024-09-18T00:42:27.726 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.Fa4umRLnwz 2024-09-18T00:42:27.914 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.Fa4umRLnwz 2024-09-18T00:42:28.006 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.Fa4umRLnwz is 603KB 2024-09-18T00:42:28.079 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.Fa4umRLnwz 2024-09-18T00:42:28.095 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-18T00:42:28.095 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T00:42:28.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T00:42:28.200 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-18T00:42:28.240 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-18T00:42:28.240 INFO:tasks.ceph:Archiving crash dumps... 2024-09-18T00:42:28.242 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909591/remote/smithi096/crash 2024-09-18T00:42:28.243 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T00:42:28.269 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909591/remote/smithi192/crash 2024-09-18T00:42:28.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T00:42:28.308 INFO:tasks.ceph:Compressing logs... 2024-09-18T00:42:28.308 DEBUG:teuthology.orchestra.run.smithi096:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T00:42:28.314 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T00:42:28.351 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46013.log 2024-09-18T00:42:28.376 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33842.log 2024-09-18T00:42:28.377 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46221.log 2024-09-18T00:42:28.377 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.46013.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46013.log.gz 2024-09-18T00:42:28.377 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-18T00:42:28.377 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33842.log.gz 2024-09-18T00:42:28.377 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.46221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33461.log 2024-09-18T00:42:28.377 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46221.log.gz 2024-09-18T00:42:28.377 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45803.log 2024-09-18T00:42:28.377 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-18T00:42:28.378 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33461.log.gz 2024-09-18T00:42:28.378 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.45803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33713.log 2024-09-18T00:42:28.378 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45803.log.gz 2024-09-18T00:42:28.378 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33884.log 2024-09-18T00:42:28.378 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33713.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45929.log 2024-09-18T00:42:28.378 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33713.log.gz 2024-09-18T00:42:28.378 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33201.log 2024-09-18T00:42:28.378 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33884.log.gz 2024-09-18T00:42:28.378 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.45929.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45887.log 2024-09-18T00:42:28.378 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45929.log.gz 2024-09-18T00:42:28.379 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29945.log 2024-09-18T00:42:28.379 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33201.log.gz 2024-09-18T00:42:28.379 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.45887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30084.log 2024-09-18T00:42:28.379 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45887.log.gz 2024-09-18T00:42:28.379 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.29945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46181.log 2024-09-18T00:42:28.379 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29945.log.gz 2024-09-18T00:42:28.379 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.30084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33587.log 2024-09-18T00:42:28.379 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30084.log.gz 2024-09-18T00:42:28.379 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.46181.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33714.log 2024-09-18T00:42:28.379 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46181.log.gz 2024-09-18T00:42:28.379 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33800.log 2024-09-18T00:42:28.381 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33587.log.gz 2024-09-18T00:42:28.381 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33714.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33548.log 2024-09-18T00:42:28.381 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33714.log.gz 2024-09-18T00:42:28.381 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32712.log 2024-09-18T00:42:28.381 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33800.log.gz 2024-09-18T00:42:28.381 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29985.log 2024-09-18T00:42:28.382 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33548.log.gz 2024-09-18T00:42:28.382 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.32712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46055.log 2024-09-18T00:42:28.382 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32712.log.gz 2024-09-18T00:42:28.382 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.29985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33332.log 2024-09-18T00:42:28.382 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29985.log.gz 2024-09-18T00:42:28.382 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.46055.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33026.log 2024-09-18T00:42:28.382 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46055.log.gz 2024-09-18T00:42:28.382 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33332.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33332.log.gz 2024-09-18T00:42:28.382 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30156.log 2024-09-18T00:42:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-18T00:42:28.383 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33026.log.gz 2024-09-18T00:42:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.30156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33421.log 2024-09-18T00:42:28.383 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30156.log.gz 2024-09-18T00:42:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33107.log 2024-09-18T00:42:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-18T00:42:28.383 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33421.log.gz 2024-09-18T00:42:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33107.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33588.log 2024-09-18T00:42:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33674.log 2024-09-18T00:42:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster2-mgr.x.log: 80.4% -- replaced with /var/log/ceph/cluster1-client.admin.33107.log.gz 2024-09-18T00:42:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33588.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33158.log 2024-09-18T00:42:28.383 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33588.log.gz 2024-09-18T00:42:28.383 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-18T00:42:28.384 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33674.log.gz 2024-09-18T00:42:28.384 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33245.log 2024-09-18T00:42:28.384 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33158.log.gz 2024-09-18T00:42:28.384 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33460.log 2024-09-18T00:42:28.384 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33245.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29747.log 2024-09-18T00:42:28.384 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33245.log.gz 2024-09-18T00:42:28.384 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45971.log 2024-09-18T00:42:28.384 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33460.log.gz 2024-09-18T00:42:28.384 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.29747.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-18T00:42:28.384 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29747.log.gz 2024-09-18T00:42:28.384 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.45971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33082.log 2024-09-18T00:42:28.384 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45971.log.gz 2024-09-18T00:42:28.385 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.25256.log 2024-09-18T00:42:28.385 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-18T00:42:28.385 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33082.log.gz 2024-09-18T00:42:28.385 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph.tmp-client.admin.25256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46139.log 2024-09-18T00:42:28.385 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.25256.log.gz 2024-09-18T00:42:28.385 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33331.log 2024-09-18T00:42:28.385 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.46139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45719.log 2024-09-18T00:42:28.385 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46139.log.gz 2024-09-18T00:42:28.385 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29905.log 2024-09-18T00:42:28.385 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33331.log.gz 2024-09-18T00:42:28.385 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.45719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33333.log 2024-09-18T00:42:28.385 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45719.log.gz 2024-09-18T00:42:28.385 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.29905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45845.log 2024-09-18T00:42:28.385 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29905.log.gz 2024-09-18T00:42:28.386 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.33333.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-18T00:42:28.386 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.33333.log.gz 2024-09-18T00:42:28.386 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.45845.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45761.log 2024-09-18T00:42:28.386 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45845.log.gz 2024-09-18T00:42:28.386 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46097.log 2024-09-18T00:42:28.386 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.45761.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.45761.log.gz 2024-09-18T00:42:28.386 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.46097.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46097.log.gz 2024-09-18T00:42:28.386 INFO:teuthology.orchestra.run.smithi096.stderr: 87.3% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-18T00:42:28.411 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.81928.log 2024-09-18T00:42:28.411 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107709.log 2024-09-18T00:42:28.419 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.81928.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129994.log 2024-09-18T00:42:28.420 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.107709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106825.log 2024-09-18T00:42:28.420 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129994.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151450.log 2024-09-18T00:42:28.420 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106825.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40389.log 2024-09-18T00:42:28.420 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151450.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56487.log 2024-09-18T00:42:28.420 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43376.log 2024-09-18T00:42:28.420 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100461.log 2024-09-18T00:42:28.420 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117679.log 2024-09-18T00:42:28.420 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100461.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46416.log 2024-09-18T00:42:28.420 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.117679.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75508.log 2024-09-18T00:42:28.420 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90431.log 2024-09-18T00:42:28.420 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46416.log.gz 2024-09-18T00:42:28.421 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75508.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69222.log 2024-09-18T00:42:28.421 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101978.log 2024-09-18T00:42:28.421 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41483.log 2024-09-18T00:42:28.421 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116226.log 2024-09-18T00:42:28.421 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109671.log 2024-09-18T00:42:28.421 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100881.log 2024-09-18T00:42:28.421 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123247.log 2024-09-18T00:42:28.421 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109671.log: /var/log/ceph/cluster1-client.mirror.100881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137175.log 2024-09-18T00:42:28.421 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119343.log 2024-09-18T00:42:28.421 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109478.log 2024-09-18T00:42:28.421 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137175.log: /var/log/ceph/cluster1-client.mirror.119343.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128692.log 2024-09-18T00:42:28.421 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109478.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50660.log 2024-09-18T00:42:28.421 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43122.log 2024-09-18T00:42:28.422 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.50660.log: /var/log/ceph/cluster1-client.mirror.150702.log 2024-09-18T00:42:28.422 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72369.log 2024-09-18T00:42:28.422 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133258.log 2024-09-18T00:42:28.422 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59163.log 2024-09-18T00:42:28.422 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113531.log 2024-09-18T00:42:28.422 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41275.log 2024-09-18T00:42:28.422 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50369.log 2024-09-18T00:42:28.422 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.41275.log: --verbose -- /var/log/ceph/cluster2-client.admin.152676.log 2024-09-18T00:42:28.434 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135010.log 2024-09-18T00:42:28.434 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60875.log 2024-09-18T00:42:28.434 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152676.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.152676.log.gz 2024-09-18T00:42:28.434 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58593.log 2024-09-18T00:42:28.434 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87374.log 2024-09-18T00:42:28.434 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.58593.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134149.log 2024-09-18T00:42:28.434 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87374.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128134.log 2024-09-18T00:42:28.434 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134149.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93789.log 2024-09-18T00:42:28.435 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138705.log 2024-09-18T00:42:28.435 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128134.log: /var/log/ceph/cluster2-client.mirror.93789.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137007.log 2024-09-18T00:42:28.435 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98319.log 2024-09-18T00:42:28.435 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93029.log 2024-09-18T00:42:28.435 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137007.log: /var/log/ceph/cluster1-client.mirror.98319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40852.log 2024-09-18T00:42:28.435 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68022.log 2024-09-18T00:42:28.435 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.40852.log: /var/log/ceph/cluster1-client.admin.56674.log 2024-09-18T00:42:28.435 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.68022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139284.log 2024-09-18T00:42:28.435 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102726.log 2024-09-18T00:42:28.435 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139284.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118135.log 2024-09-18T00:42:28.435 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster1-client.mirror.102726.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108121.log 2024-09-18T00:42:28.435 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89461.log 2024-09-18T00:42:28.436 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42232.log 2024-09-18T00:42:28.436 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35653.log 2024-09-18T00:42:28.436 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114502.log 2024-09-18T00:42:28.436 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42232.log: /var/log/ceph/cluster1-client.mirror.35653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135136.log 2024-09-18T00:42:28.436 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114502.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113451.log 2024-09-18T00:42:28.436 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130295.log 2024-09-18T00:42:28.436 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.113451.log: -- /var/log/ceph/cluster1-client.mirror.134300.log 2024-09-18T00:42:28.436 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130295.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51835.log 2024-09-18T00:42:28.436 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134300.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139932.log 2024-09-18T00:42:28.436 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77070.log 2024-09-18T00:42:28.436 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136312.log 2024-09-18T00:42:28.436 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77070.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37701.log 2024-09-18T00:42:28.436 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49004.log 2024-09-18T00:42:28.437 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136312.log: /var/log/ceph/cluster1-client.mirror.37701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111855.log 2024-09-18T00:42:28.437 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78272.log 2024-09-18T00:42:28.437 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106932.log 2024-09-18T00:42:28.437 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52846.log 2024-09-18T00:42:28.437 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80257.log 2024-09-18T00:42:28.437 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145645.log 2024-09-18T00:42:28.437 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.81928.log.gz 2024-09-18T00:42:28.437 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.80257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109390.log 2024-09-18T00:42:28.437 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64170.log 2024-09-18T00:42:28.437 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101801.log 2024-09-18T00:42:28.437 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125853.log 2024-09-18T00:42:28.437 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64170.log: /var/log/ceph/cluster1-client.mirror.101801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121419.log 2024-09-18T00:42:28.437 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109435.log 2024-09-18T00:42:28.438 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118228.log 2024-09-18T00:42:28.438 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109435.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127535.log 2024-09-18T00:42:28.438 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.118228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101780.log 2024-09-18T00:42:28.438 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster1-client.admin.127535.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77956.log 2024-09-18T00:42:28.438 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101780.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104228.log 2024-09-18T00:42:28.438 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113531.log.gz 2024-09-18T00:42:28.438 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77956.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43375.log 2024-09-18T00:42:28.438 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.104228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130257.log 2024-09-18T00:42:28.438 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88269.log 2024-09-18T00:42:28.438 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130257.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81636.log 2024-09-18T00:42:28.438 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148736.log 2024-09-18T00:42:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110175.log 2024-09-18T00:42:28.454 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81636.log.gz 2024-09-18T00:42:28.454 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83045.log 2024-09-18T00:42:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148736.log: /var/log/ceph/cluster1-client.mirror.110175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87272.log 2024-09-18T00:42:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.83045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102897.log 2024-09-18T00:42:28.455 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41483.log.gz 2024-09-18T00:42:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87272.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69648.log 2024-09-18T00:42:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102897.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62196.log 2024-09-18T00:42:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101090.log 2024-09-18T00:42:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62196.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124252.log 2024-09-18T00:42:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138593.log 2024-09-18T00:42:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103085.log 2024-09-18T00:42:28.455 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123247.log.gz 2024-09-18T00:42:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster1-client.admin.138593.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124319.log 2024-09-18T00:42:28.455 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.138593.log.gz 2024-09-18T00:42:28.455 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.103085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115429.log 2024-09-18T00:42:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54147.log 2024-09-18T00:42:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111811.log 2024-09-18T00:42:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54147.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53643.log 2024-09-18T00:42:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137575.log 2024-09-18T00:42:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125327.log 2024-09-18T00:42:28.456 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43122.log.gz 2024-09-18T00:42:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51585.log 2024-09-18T00:42:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137575.log: /var/log/ceph/cluster1-client.mirror.125327.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56692.log 2024-09-18T00:42:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51585.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125368.log 2024-09-18T00:42:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53918.log 2024-09-18T00:42:28.456 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56692.log: gzip 87.6% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.106825.log.gz --verbose 2024-09-18T00:42:28.456 INFO:teuthology.orchestra.run.smithi192.stderr: -- /var/log/ceph/cluster1-client.mirror.98738.log 2024-09-18T00:42:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125368.log: /var/log/ceph/cluster1-client.mirror.53918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.81841.log 2024-09-18T00:42:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91878.log 2024-09-18T00:42:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.81841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79678.log 2024-09-18T00:42:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69650.log 2024-09-18T00:42:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79678.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110176.log 2024-09-18T00:42:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster2-client.admin.69650.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47507.log 2024-09-18T00:42:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110176.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44370.log 2024-09-18T00:42:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.47507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42233.log 2024-09-18T00:42:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44370.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72857.log 2024-09-18T00:42:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76760.log 2024-09-18T00:42:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42233.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70127.log 2024-09-18T00:42:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72857.log: /var/log/ceph/cluster1-client.mirror.76760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47904.log 2024-09-18T00:42:28.457 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.107709.log.gz 2024-09-18T00:42:28.457 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70127.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87700.log 2024-09-18T00:42:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89298.log 2024-09-18T00:42:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131415.log 2024-09-18T00:42:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39211.log 2024-09-18T00:42:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster1-client.mirror.131415.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96389.log 2024-09-18T00:42:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60057.log 2024-09-18T00:42:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96389.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128919.log 2024-09-18T00:42:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.60057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40009.log 2024-09-18T00:42:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66648.log 2024-09-18T00:42:28.458 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128919.log.gz 2024-09-18T00:42:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53686.log 2024-09-18T00:42:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139967.log 2024-09-18T00:42:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66648.log: /var/log/ceph/cluster1-client.mirror.53686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118296.log 2024-09-18T00:42:28.458 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.139967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41696.log 2024-09-18T00:42:28.459 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.admin.118296.log: /var/log/ceph/cluster1-client.mirror.53348.log 2024-09-18T00:42:28.459 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41696.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129787.log 2024-09-18T00:42:28.459 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73971.log 2024-09-18T00:42:28.459 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53348.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39359.log 2024-09-18T00:42:28.459 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.129787.log: /var/log/ceph/cluster1-client.mirror.73971.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46306.log 2024-09-18T00:42:28.459 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39359.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74919.log 2024-09-18T00:42:28.459 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster2-client.admin.46306.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118436.log 2024-09-18T00:42:28.459 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141563.log 2024-09-18T00:42:28.459 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.118436.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48043.log 2024-09-18T00:42:28.459 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.141563.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41023.log 2024-09-18T00:42:28.459 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151423.log 2024-09-18T00:42:28.459 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106957.log 2024-09-18T00:42:28.459 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41023.log: /var/log/ceph/cluster1-client.admin.151423.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139576.log 2024-09-18T00:42:28.459 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43376.log.gz 2024-09-18T00:42:28.460 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49114.log 2024-09-18T00:42:28.460 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.139576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64800.log 2024-09-18T00:42:28.460 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49114.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41906.log 2024-09-18T00:42:28.460 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105463.log 2024-09-18T00:42:28.460 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64800.log: /var/log/ceph/cluster1-client.mirror.41906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135559.log 2024-09-18T00:42:28.460 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.105463.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141408.log 2024-09-18T00:42:28.460 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105463.log.gz 2024-09-18T00:42:28.460 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster1-client.mirror.135559.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.56186.log 2024-09-18T00:42:28.460 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.116226.log.gz 2024-09-18T00:42:28.460 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141408.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106712.log 2024-09-18T00:42:28.460 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56186.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82592.log 2024-09-18T00:42:28.460 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.56186.log.gz 2024-09-18T00:42:28.460 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.106712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83825.log 2024-09-18T00:42:28.461 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.82592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103217.log 2024-09-18T00:42:28.461 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83825.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52592.log 2024-09-18T00:42:28.461 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.103217.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111012.log 2024-09-18T00:42:28.461 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146447.log 2024-09-18T00:42:28.461 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52592.log: /var/log/ceph/cluster1-client.mirror.111012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54860.log 2024-09-18T00:42:28.461 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136166.log 2024-09-18T00:42:28.500 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54860.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113328.log 2024-09-18T00:42:28.500 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136166.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116618.log 2024-09-18T00:42:28.500 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87273.log 2024-09-18T00:42:28.500 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.117679.log.gz 2024-09-18T00:42:28.500 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.146447.log.gz 2024-09-18T00:42:28.500 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.116618.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122218.log 2024-09-18T00:42:28.500 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63940.log 2024-09-18T00:42:28.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146508.log 2024-09-18T00:42:28.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112862.log 2024-09-18T00:42:28.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146508.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141677.log 2024-09-18T00:42:28.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112862.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123330.log 2024-09-18T00:42:28.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.141677.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120251.log 2024-09-18T00:42:28.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111016.log 2024-09-18T00:42:28.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102056.log 2024-09-18T00:42:28.501 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.146508.log.gz 2024-09-18T00:42:28.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111016.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60764.log 2024-09-18T00:42:28.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71510.log 2024-09-18T00:42:28.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60764.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76005.log 2024-09-18T00:42:28.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141013.log 2024-09-18T00:42:28.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76005.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102350.log 2024-09-18T00:42:28.502 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127130.log 2024-09-18T00:42:28.502 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76905.log 2024-09-18T00:42:28.502 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142581.log 2024-09-18T00:42:28.502 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62826.log 2024-09-18T00:42:28.502 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76905.log: /var/log/ceph/cluster1-client.mirror.142581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110302.log 2024-09-18T00:42:28.502 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96409.log 2024-09-18T00:42:28.502 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62826.log: gzip/var/log/ceph/cluster1-client.mirror.110302.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36623.log 2024-09-18T00:42:28.502 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50369.log.gz 2024-09-18T00:42:28.502 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98528.log 2024-09-18T00:42:28.502 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36623.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99956.log 2024-09-18T00:42:28.502 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98528.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137867.log 2024-09-18T00:42:28.502 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99956.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89589.log 2024-09-18T00:42:28.502 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.137867.log: --verbose -- /var/log/ceph/cluster2-client.mirror.93113.log 2024-09-18T00:42:28.503 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95724.log 2024-09-18T00:42:28.503 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.93113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78670.log 2024-09-18T00:42:28.503 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8%/var/log/ceph/cluster1-client.mirror.95724.log: -- replaced with /var/log/ceph/cluster1-client.mirror.89461.log.gz 2024-09-18T00:42:28.503 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122090.log 2024-09-18T00:42:28.503 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86384.log 2024-09-18T00:42:28.503 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138013.log 2024-09-18T00:42:28.503 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88311.log 2024-09-18T00:42:28.503 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63500.log 2024-09-18T00:42:28.503 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88311.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91697.log 2024-09-18T00:42:28.503 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80976.log 2024-09-18T00:42:28.503 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63500.log: gzip -5/var/log/ceph/cluster1-client.admin.91697.log: --verbose -- /var/log/ceph/cluster1-client.mirror.134717.log 2024-09-18T00:42:28.503 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149872.log 2024-09-18T00:42:28.504 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30525.log 2024-09-18T00:42:28.504 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134717.log: /var/log/ceph/cluster1-client.mirror.149872.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75170.log 2024-09-18T00:42:28.504 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50660.log.gz 2024-09-18T00:42:28.504 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99890.log 2024-09-18T00:42:28.504 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.30525.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30525.log.gz 2024-09-18T00:42:28.504 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75170.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99620.log 2024-09-18T00:42:28.504 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103379.log 2024-09-18T00:42:28.504 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99620.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107526.log 2024-09-18T00:42:28.504 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.103379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76195.log 2024-09-18T00:42:28.504 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114542.log 2024-09-18T00:42:28.504 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107526.log: /var/log/ceph/cluster1-client.mirror.76195.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136315.log 2024-09-18T00:42:28.504 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.114542.log: -- /var/log/ceph/cluster1-client.admin.44886.log 2024-09-18T00:42:28.504 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136315.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142229.log 2024-09-18T00:42:28.505 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108121.log.gz 2024-09-18T00:42:28.505 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99409.log 2024-09-18T00:42:28.505 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142229.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115600.log 2024-09-18T00:42:28.505 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47185.log 2024-09-18T00:42:28.505 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115600.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54399.log 2024-09-18T00:42:28.505 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72321.log 2024-09-18T00:42:28.505 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64171.log 2024-09-18T00:42:28.505 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.72321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97584.log 2024-09-18T00:42:28.505 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64171.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36621.log 2024-09-18T00:42:28.505 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.97584.log: -- /var/log/ceph/cluster1-client.mirror.86100.log 2024-09-18T00:42:28.505 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster1-client.mirror.36621.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118208.log 2024-09-18T00:42:28.505 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100881.log.gz 2024-09-18T00:42:28.505 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89949.log 2024-09-18T00:42:28.506 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.118208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122301.log 2024-09-18T00:42:28.506 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75131.log 2024-09-18T00:42:28.506 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102057.log 2024-09-18T00:42:28.506 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122408.log 2024-09-18T00:42:28.506 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127644.log 2024-09-18T00:42:28.506 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.122408.log: --verbose -- /var/log/ceph/cluster1-client.admin.150503.log 2024-09-18T00:42:28.506 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102726.log.gz 2024-09-18T00:42:28.506 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.127644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89740.log 2024-09-18T00:42:28.506 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76390.log 2024-09-18T00:42:28.506 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95572.log 2024-09-18T00:42:28.506 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75211.log 2024-09-18T00:42:28.506 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136314.log 2024-09-18T00:42:28.506 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78736.log 2024-09-18T00:42:28.507 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75211.log: /var/log/ceph/cluster1-client.mirror.136314.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81623.log 2024-09-18T00:42:28.507 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69922.log 2024-09-18T00:42:28.507 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81623.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58499.log 2024-09-18T00:42:28.507 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81623.log.gz 2024-09-18T00:42:28.507 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69922.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87208.log 2024-09-18T00:42:28.507 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.69922.log.gz 2024-09-18T00:42:28.507 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.58499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82136.log 2024-09-18T00:42:28.507 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87208.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92910.log 2024-09-18T00:42:28.507 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.82136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44731.log 2024-09-18T00:42:28.507 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102896.log 2024-09-18T00:42:28.507 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69290.log 2024-09-18T00:42:28.507 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102896.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137446.log 2024-09-18T00:42:28.507 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.69290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36929.log 2024-09-18T00:42:28.508 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90431.log.gz 2024-09-18T00:42:28.508 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137446.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102476.log 2024-09-18T00:42:28.508 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36929.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102222.log 2024-09-18T00:42:28.508 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44442.log 2024-09-18T00:42:28.508 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138741.log 2024-09-18T00:42:28.508 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79979.log 2024-09-18T00:42:28.508 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153722.log 2024-09-18T00:42:28.508 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.138741.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.138741.log.gz 2024-09-18T00:42:28.508 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41813.log 2024-09-18T00:42:28.508 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.153722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153286.log 2024-09-18T00:42:28.508 INFO:teuthology.orchestra.run.smithi192.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.153722.log.gz 2024-09-18T00:42:28.508 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100461.log.gz 2024-09-18T00:42:28.508 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64380.log 2024-09-18T00:42:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.153286.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52027.log 2024-09-18T00:42:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64380.log: 87.3%gzip -- replaced with /var/log/ceph/cluster1-client.admin.153286.log.gz 2024-09-18T00:42:28.509 INFO:teuthology.orchestra.run.smithi192.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62615.log 2024-09-18T00:42:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52027.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81794.log 2024-09-18T00:42:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46843.log 2024-09-18T00:42:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.81794.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83233.log 2024-09-18T00:42:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.46843.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38728.log 2024-09-18T00:42:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83233.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72649.log 2024-09-18T00:42:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110511.log 2024-09-18T00:42:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72649.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136819.log 2024-09-18T00:42:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137174.log 2024-09-18T00:42:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128133.log 2024-09-18T00:42:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136819.log: gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.137174.log: /var/log/ceph/cluster1-client.mirror.131325.log 2024-09-18T00:42:28.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128133.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151153.log 2024-09-18T00:42:28.510 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133258.log.gz 2024-09-18T00:42:28.510 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127221.log 2024-09-18T00:42:28.510 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136189.log 2024-09-18T00:42:28.510 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139055.log 2024-09-18T00:42:28.510 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136189.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38180.log 2024-09-18T00:42:28.510 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139055.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70557.log 2024-09-18T00:42:28.510 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38180.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77872.log 2024-09-18T00:42:28.510 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35232.log 2024-09-18T00:42:28.510 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster1-client.mirror.77872.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106251.log 2024-09-18T00:42:28.511 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.35232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112697.log 2024-09-18T00:42:28.511 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.106251.log: /var/log/ceph/cluster1-client.admin.83472.log 2024-09-18T00:42:28.511 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112697.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103470.log 2024-09-18T00:42:28.511 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.150702.log.gz 2024-09-18T00:42:28.511 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142645.log 2024-09-18T00:42:28.511 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.83472.log: gzip -5 --verbose/var/log/ceph/cluster1-client.admin.103470.log: -- /var/log/ceph/cluster1-client.mirror.63624.log 2024-09-18T00:42:28.511 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.103470.log.gz 2024-09-18T00:42:28.511 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54230.log 2024-09-18T00:42:28.511 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142645.log: /var/log/ceph/cluster1-client.mirror.63624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138291.log 2024-09-18T00:42:28.511 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54230.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89152.log 2024-09-18T00:42:28.511 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95249.log 2024-09-18T00:42:28.511 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64465.log 2024-09-18T00:42:28.511 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89152.log: /var/log/ceph/cluster1-client.mirror.95249.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85740.log 2024-09-18T00:42:28.512 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96013.log 2024-09-18T00:42:28.512 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85740.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77070.log.gz 2024-09-18T00:42:28.512 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40390.log 2024-09-18T00:42:28.512 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96013.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105205.log 2024-09-18T00:42:28.512 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109922.log 2024-09-18T00:42:28.512 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.105205.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102562.log 2024-09-18T00:42:28.512 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109922.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62448.log 2024-09-18T00:42:28.512 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107653.log 2024-09-18T00:42:28.512 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102562.log: /var/log/ceph/cluster1-client.mirror.62448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94313.log 2024-09-18T00:42:28.512 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107653.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45696.log 2024-09-18T00:42:28.512 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.94313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84129.log 2024-09-18T00:42:28.512 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45696.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59183.log 2024-09-18T00:42:28.513 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58119.log 2024-09-18T00:42:28.513 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.59183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129693.log 2024-09-18T00:42:28.513 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61488.log 2024-09-18T00:42:28.513 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.58119.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.58119.log.gz 2024-09-18T00:42:28.513 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115786.log 2024-09-18T00:42:28.513 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.61488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123224.log 2024-09-18T00:42:28.513 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.admin.115786.log: /var/log/ceph/cluster1-client.mirror.97607.log 2024-09-18T00:42:28.513 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74581.log 2024-09-18T00:42:28.513 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109671.log.gz 2024-09-18T00:42:28.514 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97607.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129606.log 2024-09-18T00:42:28.514 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38727.log 2024-09-18T00:42:28.514 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146446.log 2024-09-18T00:42:28.514 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149363.log 2024-09-18T00:42:28.514 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38727.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98319.log.gz 2024-09-18T00:42:28.514 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146446.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150691.log 2024-09-18T00:42:28.514 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149363.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130129.log 2024-09-18T00:42:28.514 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150691.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.150691.log.gz 2024-09-18T00:42:28.514 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64131.log 2024-09-18T00:42:28.514 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141314.log 2024-09-18T00:42:28.514 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130129.log: /var/log/ceph/cluster1-client.mirror.64131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38057.log 2024-09-18T00:42:28.514 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40019.log 2024-09-18T00:42:28.514 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110091.log 2024-09-18T00:42:28.515 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134339.log 2024-09-18T00:42:28.515 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101847.log 2024-09-18T00:42:28.515 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102432.log 2024-09-18T00:42:28.515 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101847.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138225.log 2024-09-18T00:42:28.515 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135641.log 2024-09-18T00:42:28.515 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52676.log 2024-09-18T00:42:28.515 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.87374.log.gz 2024-09-18T00:42:28.515 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145963.log 2024-09-18T00:42:28.515 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109881.log 2024-09-18T00:42:28.515 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107064.log 2024-09-18T00:42:28.515 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93037.log 2024-09-18T00:42:28.515 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103327.log 2024-09-18T00:42:28.515 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5/var/log/ceph/cluster2-client.admin.93037.log: --verbose -- /var/log/ceph/cluster2-client.admin.105450.log 2024-09-18T00:42:28.516 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.103327.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44607.log 2024-09-18T00:42:28.516 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster2-client.admin.105450.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124696.log 2024-09-18T00:42:28.516 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105450.log.gz 2024-09-18T00:42:28.516 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44607.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104107.log 2024-09-18T00:42:28.516 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124696.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101911.log 2024-09-18T00:42:28.516 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104107.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118496.log 2024-09-18T00:42:28.516 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.101911.log: -- /var/log/ceph/cluster1-client.mirror.126460.log 2024-09-18T00:42:28.516 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101978.log.gz 2024-09-18T00:42:28.516 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110171.log 2024-09-18T00:42:28.516 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125660.log 2024-09-18T00:42:28.516 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110171.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98318.log 2024-09-18T00:42:28.543 INFO:teuthology.orchestra.run.smithi096.stderr: 93.5% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-18T00:42:28.670 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125660.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116224.log 2024-09-18T00:42:28.670 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98318.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72619.log 2024-09-18T00:42:28.670 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115129.log 2024-09-18T00:42:28.670 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116224.log: /var/log/ceph/cluster1-client.mirror.72619.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39151.log 2024-09-18T00:42:28.670 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster1-client.admin.115129.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47772.log 2024-09-18T00:42:28.671 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.115129.log.gz 2024-09-18T00:42:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39151.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99913.log 2024-09-18T00:42:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47642.log 2024-09-18T00:42:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.99913.log: --verbose -- /var/log/ceph/cluster1-client.mirror.39589.log 2024-09-18T00:42:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85934.log 2024-09-18T00:42:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46634.log 2024-09-18T00:42:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126080.log 2024-09-18T00:42:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.46634.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58613.log 2024-09-18T00:42:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126080.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52847.log 2024-09-18T00:42:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.58613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106896.log 2024-09-18T00:42:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52847.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134970.log 2024-09-18T00:42:28.671 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106896.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87141.log 2024-09-18T00:42:28.672 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145551.log 2024-09-18T00:42:28.672 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147263.log 2024-09-18T00:42:28.672 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87141.log: /var/log/ceph/cluster1-client.mirror.145551.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.109478.log.gz 2024-09-18T00:42:28.672 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69269.log 2024-09-18T00:42:28.672 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139730.log 2024-09-18T00:42:28.672 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58640.log 2024-09-18T00:42:28.672 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139730.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90812.log 2024-09-18T00:42:28.672 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127993.log 2024-09-18T00:42:28.672 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.58640.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.147263.log.gz 2024-09-18T00:42:28.672 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90812.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51710.log 2024-09-18T00:42:28.672 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.127993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91977.log 2024-09-18T00:42:28.672 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51710.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92483.log 2024-09-18T00:42:28.672 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101913.log 2024-09-18T00:42:28.673 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92483.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104507.log 2024-09-18T00:42:28.673 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101913.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99431.log 2024-09-18T00:42:28.673 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126923.log 2024-09-18T00:42:28.673 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.104507.log: /var/log/ceph/cluster1-client.mirror.99431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55135.log 2024-09-18T00:42:28.673 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.126923.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100818.log 2024-09-18T00:42:28.673 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.126923.log.gz 2024-09-18T00:42:28.673 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61839.log 2024-09-18T00:42:28.673 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110932.log 2024-09-18T00:42:28.673 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.61839.log: --verbose -- /var/log/ceph/cluster1-client.mirror.53265.log 2024-09-18T00:42:28.673 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79004.log 2024-09-18T00:42:28.673 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster1-client.mirror.53265.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58780.log 2024-09-18T00:42:28.673 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62786.log 2024-09-18T00:42:28.674 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.58780.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43794.log 2024-09-18T00:42:28.674 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62786.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50344.log 2024-09-18T00:42:28.674 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43794.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135220.log 2024-09-18T00:42:28.674 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50344.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38663.log 2024-09-18T00:42:28.674 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42114.log 2024-09-18T00:42:28.674 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152332.log 2024-09-18T00:42:28.674 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38663.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106896.log.gz 2024-09-18T00:42:28.674 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42114.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59405.log 2024-09-18T00:42:28.674 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102519.log 2024-09-18T00:42:28.674 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40389.log.gz 2024-09-18T00:42:28.674 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.152332.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106932.log.gz 2024-09-18T00:42:28.674 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59405.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82384.log 2024-09-18T00:42:28.674 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45090.log 2024-09-18T00:42:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.82384.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140315.log 2024-09-18T00:42:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.45090.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34819.log 2024-09-18T00:42:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140315.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49469.log 2024-09-18T00:42:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.34819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55448.log 2024-09-18T00:42:28.675 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34819.log.gz 2024-09-18T00:42:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98048.log 2024-09-18T00:42:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49469.log: /var/log/ceph/cluster1-client.mirror.55448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56085.log 2024-09-18T00:42:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98048.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104021.log 2024-09-18T00:42:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139354.log 2024-09-18T00:42:28.675 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.56085.log.gz 2024-09-18T00:42:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125241.log 2024-09-18T00:42:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45335.log 2024-09-18T00:42:28.675 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43800.log 2024-09-18T00:42:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98361.log 2024-09-18T00:42:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74520.log 2024-09-18T00:42:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster1-client.mirror.98361.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69403.log 2024-09-18T00:42:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74520.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104854.log 2024-09-18T00:42:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.69403.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127538.log 2024-09-18T00:42:28.676 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41275.log.gz 2024-09-18T00:42:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104854.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62472.log 2024-09-18T00:42:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster1-client.admin.127538.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124822.log 2024-09-18T00:42:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62472.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51374.log 2024-09-18T00:42:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124822.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106385.log 2024-09-18T00:42:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48284.log 2024-09-18T00:42:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster2-client.mirror.106385.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63541.log 2024-09-18T00:42:28.676 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.48284.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128958.log 2024-09-18T00:42:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40132.log 2024-09-18T00:42:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128958.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95061.log 2024-09-18T00:42:28.677 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128958.log.gz 2024-09-18T00:42:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56682.log 2024-09-18T00:42:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.95061.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106593.log 2024-09-18T00:42:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56682.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103065.log 2024-09-18T00:42:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.154140.log 2024-09-18T00:42:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.106593.log: /var/log/ceph/cluster1-client.mirror.103065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87825.log 2024-09-18T00:42:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.154140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139628.log 2024-09-18T00:42:28.677 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.154140.log.gz 2024-09-18T00:42:28.677 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.79979.log.gz 2024-09-18T00:42:28.677 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87825.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45258.log 2024-09-18T00:42:28.678 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139628.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47857.log 2024-09-18T00:42:28.678 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54882.log 2024-09-18T00:42:28.678 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.47857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85170.log 2024-09-18T00:42:28.678 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42747.log 2024-09-18T00:42:28.678 INFO:teuthology.orchestra.run.smithi192.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.118135.log.gz 2024-09-18T00:42:28.678 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65683.log 2024-09-18T00:42:28.678 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85170.log: /var/log/ceph/cluster1-client.mirror.42747.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76650.log 2024-09-18T00:42:28.678 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65683.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88709.log 2024-09-18T00:42:28.678 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76650.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115934.log 2024-09-18T00:42:28.678 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48109.log 2024-09-18T00:42:28.678 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.115934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141268.log 2024-09-18T00:42:28.678 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48109.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112902.log 2024-09-18T00:42:28.678 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.admin.141268.log: -- /var/log/ceph/cluster1-client.mirror.110595.log 2024-09-18T00:42:28.678 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112902.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46008.log 2024-09-18T00:42:28.679 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110595.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54568.log 2024-09-18T00:42:28.679 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52930.log 2024-09-18T00:42:28.679 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.130295.log.gz 2024-09-18T00:42:28.679 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102979.log 2024-09-18T00:42:28.679 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54568.log: /var/log/ceph/cluster1-client.mirror.52930.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52675.log 2024-09-18T00:42:28.679 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102979.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57023.log 2024-09-18T00:42:28.679 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105204.log 2024-09-18T00:42:28.679 INFO:teuthology.orchestra.run.smithi192.stderr: 66.3% -- replaced with /var/log/ceph/cluster1-client.mirror.141563.log.gz 2024-09-18T00:42:28.679 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57023.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78415.log 2024-09-18T00:42:28.679 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.105204.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.69987.log 2024-09-18T00:42:28.679 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151168.log 2024-09-18T00:42:28.679 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.69987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46010.log 2024-09-18T00:42:28.679 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97922.log 2024-09-18T00:42:28.680 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96822.log 2024-09-18T00:42:28.680 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138457.log 2024-09-18T00:42:28.680 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97922.log: /var/log/ceph/cluster1-client.mirror.96822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87910.log 2024-09-18T00:42:28.680 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.116618.log.gz 2024-09-18T00:42:28.680 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138457.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76064.log 2024-09-18T00:42:28.680 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120905.log 2024-09-18T00:42:28.680 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99850.log 2024-09-18T00:42:28.680 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123415.log 2024-09-18T00:42:28.680 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148505.log 2024-09-18T00:42:28.680 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56482.log 2024-09-18T00:42:28.680 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138079.log 2024-09-18T00:42:28.680 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134273.log 2024-09-18T00:42:28.680 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138079.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68236.log 2024-09-18T00:42:28.681 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153015.log 2024-09-18T00:42:28.681 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.139967.log.gz 2024-09-18T00:42:28.681 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68236.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129579.log 2024-09-18T00:42:28.681 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.153015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56354.log 2024-09-18T00:42:28.681 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49025.log 2024-09-18T00:42:28.681 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56354.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66525.log 2024-09-18T00:42:28.681 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93716.log 2024-09-18T00:42:28.681 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49025.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.151866.log 2024-09-18T00:42:28.681 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66525.log: /var/log/ceph/cluster1-client.admin.93716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92006.log 2024-09-18T00:42:28.681 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.151866.log: gzip -5 --verbose 90.0% -- /var/log/ceph/cluster1-client.mirror.123770.log -- replaced with /var/log/ceph/cluster1-client.mirror.122301.log.gz 2024-09-18T00:42:28.681 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:28.681 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98150.log 2024-09-18T00:42:28.681 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92006.log: /var/log/ceph/cluster1-client.mirror.123770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66314.log 2024-09-18T00:42:28.682 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98150.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129860.log 2024-09-18T00:42:28.682 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129237.log 2024-09-18T00:42:28.682 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124319.log.gz 2024-09-18T00:42:28.682 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129860.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43913.log 2024-09-18T00:42:28.682 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101802.log 2024-09-18T00:42:28.682 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129237.log: /var/log/ceph/cluster1-client.mirror.43913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140654.log 2024-09-18T00:42:28.682 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43752.log 2024-09-18T00:42:28.682 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101802.log: /var/log/ceph/cluster2-client.admin.140654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47678.log 2024-09-18T00:42:28.682 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140654.log.gz 2024-09-18T00:42:28.682 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144784.log 2024-09-18T00:42:28.682 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47678.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.153015.log.gz 2024-09-18T00:42:28.682 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75940.log 2024-09-18T00:42:28.683 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144784.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38939.log 2024-09-18T00:42:28.683 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137805.log 2024-09-18T00:42:28.683 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38939.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153085.log 2024-09-18T00:42:28.683 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60849.log 2024-09-18T00:42:28.683 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137805.log: /var/log/ceph/cluster1-client.admin.153085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94608.log 2024-09-18T00:42:28.683 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52257.log 2024-09-18T00:42:28.683 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.94608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134426.log 2024-09-18T00:42:28.683 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98486.log 2024-09-18T00:42:28.683 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148423.log 2024-09-18T00:42:28.683 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.153085.log.gz 2024-09-18T00:42:28.683 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98486.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44102.log 2024-09-18T00:42:28.683 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41022.log 2024-09-18T00:42:28.683 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148423.log: /var/log/ceph/cluster1-client.mirror.44102.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79046.log 2024-09-18T00:42:28.684 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64128.log 2024-09-18T00:42:28.684 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79046.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42232.log.gzgzip 2024-09-18T00:42:28.684 INFO:teuthology.orchestra.run.smithi192.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96518.log 2024-09-18T00:42:28.684 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99746.log 2024-09-18T00:42:28.684 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121880.log 2024-09-18T00:42:28.684 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70966.log 2024-09-18T00:42:28.684 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142429.log 2024-09-18T00:42:28.684 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.70966.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134633.log 2024-09-18T00:42:28.684 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54104.log 2024-09-18T00:42:28.684 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129062.log 2024-09-18T00:42:28.684 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54104.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146276.log 2024-09-18T00:42:28.684 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.91697.log.gz 2024-09-18T00:42:28.684 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119868.log 2024-09-18T00:42:28.685 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146276.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152787.log 2024-09-18T00:42:28.685 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98948.log 2024-09-18T00:42:28.685 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.152787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133515.log 2024-09-18T00:42:28.685 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98948.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124190.log 2024-09-18T00:42:28.685 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.146276.log.gz 2024-09-18T00:42:28.685 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111392.log 2024-09-18T00:42:28.685 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133515.log: /var/log/ceph/cluster1-client.mirror.124190.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115616.log 2024-09-18T00:42:28.685 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111392.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149784.log 2024-09-18T00:42:28.685 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67553.log 2024-09-18T00:42:28.685 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.115616.log: /var/log/ceph/cluster1-client.mirror.149784.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42535.log 2024-09-18T00:42:28.685 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51835.log.gz 2024-09-18T00:42:28.685 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118269.log 2024-09-18T00:42:28.685 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67553.log: /var/log/ceph/cluster1-client.mirror.42535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139035.log 2024-09-18T00:42:28.686 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137660.log 2024-09-18T00:42:28.686 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4%/var/log/ceph/cluster1-client.admin.118269.log: -- replaced with /var/log/ceph/cluster2-client.admin.92910.log.gz 2024-09-18T00:42:28.686 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139035.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109542.log 2024-09-18T00:42:28.686 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137660.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116993.log 2024-09-18T00:42:28.686 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54296.log 2024-09-18T00:42:28.686 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.116993.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115807.log 2024-09-18T00:42:28.686 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54296.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51522.log 2024-09-18T00:42:28.686 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.115807.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101006.log 2024-09-18T00:42:28.686 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51522.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60017.log 2024-09-18T00:42:28.686 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101006.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60474.log 2024-09-18T00:42:28.686 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.152787.log.gz 2024-09-18T00:42:28.686 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152881.log 2024-09-18T00:42:28.686 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60017.log: /var/log/ceph/cluster1-client.mirror.60474.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101257.log 2024-09-18T00:42:28.687 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.152881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139326.log 2024-09-18T00:42:28.687 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101257.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.152881.log.gz 2024-09-18T00:42:28.687 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150208.log 2024-09-18T00:42:28.687 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52299.log 2024-09-18T00:42:28.687 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139326.log: /var/log/ceph/cluster1-client.mirror.150208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132877.log 2024-09-18T00:42:28.687 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89089.log 2024-09-18T00:42:28.687 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.132877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42745.log 2024-09-18T00:42:28.687 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129378.log 2024-09-18T00:42:28.687 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42745.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48241.log 2024-09-18T00:42:28.687 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129378.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63750.log 2024-09-18T00:42:28.687 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151169.log 2024-09-18T00:42:28.687 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121160.log 2024-09-18T00:42:28.687 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151169.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60875.log.gz 2024-09-18T00:42:28.688 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75361.log 2024-09-18T00:42:28.688 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121160.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94199.log 2024-09-18T00:42:28.688 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119610.log 2024-09-18T00:42:28.688 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.94199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99326.log 2024-09-18T00:42:28.688 INFO:teuthology.orchestra.run.smithi192.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59163.log.gz 2024-09-18T00:42:28.688 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39209.log 2024-09-18T00:42:28.688 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99326.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118322.log 2024-09-18T00:42:28.688 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51375.log 2024-09-18T00:42:28.688 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.118322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139029.log 2024-09-18T00:42:28.688 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118739.log 2024-09-18T00:42:28.688 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139029.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81194.log 2024-09-18T00:42:28.688 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70416.log 2024-09-18T00:42:28.688 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.81194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.140926.log 2024-09-18T00:42:28.689 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78795.log 2024-09-18T00:42:28.689 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.140926.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152022.log 2024-09-18T00:42:28.689 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78795.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48673.log 2024-09-18T00:42:28.689 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151485.log 2024-09-18T00:42:28.689 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48673.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93092.log 2024-09-18T00:42:28.689 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138705.log.gz 2024-09-18T00:42:28.689 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151485.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105836.log 2024-09-18T00:42:28.689 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69288.log/var/log/ceph/cluster2-client.admin.93092.log: 2024-09-18T00:42:28.689 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.105836.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147745.log 2024-09-18T00:42:28.689 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.69288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86814.log 2024-09-18T00:42:28.689 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147745.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137661.log 2024-09-18T00:42:28.689 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123416.log 2024-09-18T00:42:28.689 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101172.log 2024-09-18T00:42:28.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123416.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104935.log 2024-09-18T00:42:28.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57306.log 2024-09-18T00:42:28.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104935.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62239.log 2024-09-18T00:42:28.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130584.log 2024-09-18T00:42:28.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53223.log 2024-09-18T00:42:28.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85002.log 2024-09-18T00:42:28.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61836.log 2024-09-18T00:42:28.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145543.log 2024-09-18T00:42:28.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.61836.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100963.log 2024-09-18T00:42:28.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145543.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62367.log 2024-09-18T00:42:28.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61918.log 2024-09-18T00:42:28.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40433.log 2024-09-18T00:42:28.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.61918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145482.log 2024-09-18T00:42:28.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50227.log 2024-09-18T00:42:28.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75212.log 2024-09-18T00:42:28.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53013.log 2024-09-18T00:42:28.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129196.log 2024-09-18T00:42:28.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64026.log 2024-09-18T00:42:28.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129196.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129217.log 2024-09-18T00:42:28.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64424.log 2024-09-18T00:42:28.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.129217.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108760.log 2024-09-18T00:42:28.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67364.log 2024-09-18T00:42:28.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108760.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153567.log 2024-09-18T00:42:28.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42426.log 2024-09-18T00:42:28.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.153567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79792.log 2024-09-18T00:42:28.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58894.log 2024-09-18T00:42:28.691 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.153567.log.gz 2024-09-18T00:42:28.692 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106800.log 2024-09-18T00:42:28.692 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.79792.log.gz 2024-09-18T00:42:28.692 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.58894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89739.log 2024-09-18T00:42:28.692 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150619.log 2024-09-18T00:42:28.692 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89739.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140858.log 2024-09-18T00:42:28.692 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150619.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.150619.log.gz 2024-09-18T00:42:28.692 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66817.log 2024-09-18T00:42:28.692 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.140858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148171.log 2024-09-18T00:42:28.692 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66817.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128945.log 2024-09-18T00:42:28.692 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148171.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55365.log 2024-09-18T00:42:28.692 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114988.log 2024-09-18T00:42:28.692 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128945.log.gz 2024-09-18T00:42:28.692 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89383.log 2024-09-18T00:42:28.693 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62532.log 2024-09-18T00:42:28.693 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113491.log 2024-09-18T00:42:28.693 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144506.log 2024-09-18T00:42:28.693 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113491.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136774.log 2024-09-18T00:42:28.693 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77915.log 2024-09-18T00:42:28.693 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99327.log 2024-09-18T00:42:28.693 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77915.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41907.log 2024-09-18T00:42:28.693 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99327.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108204.log 2024-09-18T00:42:28.693 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79155.log 2024-09-18T00:42:28.693 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141983.log 2024-09-18T00:42:28.693 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108204.log: /var/log/ceph/cluster1-client.mirror.79155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39631.log 2024-09-18T00:42:28.693 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141983.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127079.log 2024-09-18T00:42:28.694 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39631.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103587.log 2024-09-18T00:42:28.694 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.119343.log.gz 2024-09-18T00:42:28.694 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.127079.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33724.log 2024-09-18T00:42:28.694 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103426.log 2024-09-18T00:42:28.694 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90281.log 2024-09-18T00:42:28.694 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33724.log.gz 2024-09-18T00:42:28.694 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.103426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105026.log 2024-09-18T00:42:28.694 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90281.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56144.log 2024-09-18T00:42:28.694 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.105026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96906.log 2024-09-18T00:42:28.694 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.56144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49159.log 2024-09-18T00:42:28.694 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43283.log 2024-09-18T00:42:28.694 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136545.log 2024-09-18T00:42:28.694 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109755.log 2024-09-18T00:42:28.695 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74794.log 2024-09-18T00:42:28.695 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103602.log 2024-09-18T00:42:28.695 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74794.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145123.log 2024-09-18T00:42:28.695 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68022.log.gz 2024-09-18T00:42:28.695 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50577.log 2024-09-18T00:42:28.695 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145123.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83129.log 2024-09-18T00:42:28.695 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.145551.log.gz 2024-09-18T00:42:28.695 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118094.log 2024-09-18T00:42:28.695 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83129.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118775.log 2024-09-18T00:42:28.695 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.118094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90603.log 2024-09-18T00:42:28.695 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118775.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52385.log 2024-09-18T00:42:28.695 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79301.log 2024-09-18T00:42:28.695 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60321.log 2024-09-18T00:42:28.696 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134171.log 2024-09-18T00:42:28.696 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.106251.log.gz 2024-09-18T00:42:28.696 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146324.log 2024-09-18T00:42:28.696 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134171.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105628.log 2024-09-18T00:42:28.696 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132156.log 2024-09-18T00:42:28.696 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.105628.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124863.log 2024-09-18T00:42:28.696 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.132156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101216.log 2024-09-18T00:42:28.696 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124863.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109975.log 2024-09-18T00:42:28.696 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101216.log: 90.0%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.101913.log.gz 2024-09-18T00:42:28.696 INFO:teuthology.orchestra.run.smithi192.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89062.log 2024-09-18T00:42:28.696 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102351.log 2024-09-18T00:42:28.696 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67365.log 2024-09-18T00:42:28.696 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83861.log 2024-09-18T00:42:28.697 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105514.log 2024-09-18T00:42:28.697 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83861.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110591.log 2024-09-18T00:42:28.697 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.105514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133395.log 2024-09-18T00:42:28.697 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110591.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48943.log 2024-09-18T00:42:28.697 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133395.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106670.log 2024-09-18T00:42:28.697 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97607.log.gz 2024-09-18T00:42:28.697 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149028.log 2024-09-18T00:42:28.697 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66858.log 2024-09-18T00:42:28.697 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106670.log: /var/log/ceph/cluster1-client.mirror.149028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134087.log 2024-09-18T00:42:28.697 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95688.log 2024-09-18T00:42:28.697 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134087.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36047.log 2024-09-18T00:42:28.697 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95688.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88036.log 2024-09-18T00:42:28.697 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36047.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77812.log 2024-09-18T00:42:28.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88036.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54608.log 2024-09-18T00:42:28.698 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.128692.log.gz 2024-09-18T00:42:28.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77812.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70469.log 2024-09-18T00:42:28.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45041.log 2024-09-18T00:42:28.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70469.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59955.log 2024-09-18T00:42:28.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.45041.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37207.log 2024-09-18T00:42:28.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91815.log 2024-09-18T00:42:28.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.37207.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74961.log 2024-09-18T00:42:28.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115421.log 2024-09-18T00:42:28.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118871.log 2024-09-18T00:42:28.698 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45461.log 2024-09-18T00:42:28.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115421.log: /var/log/ceph/cluster1-client.mirror.118871.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127058.log 2024-09-18T00:42:28.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.45461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80179.log 2024-09-18T00:42:28.699 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127058.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48394.log 2024-09-18T00:42:28.699 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.127058.log.gz 2024-09-18T00:42:28.699 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.80179.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92657.log 2024-09-18T00:42:28.699 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48394.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116850.log 2024-09-18T00:42:28.699 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92657.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84641.log 2024-09-18T00:42:28.699 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136819.log.gz 2024-09-18T00:42:28.699 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.116850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64548.log 2024-09-18T00:42:28.699 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89748.log 2024-09-18T00:42:28.699 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149157.log 2024-09-18T00:42:28.699 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84436.log 2024-09-18T00:42:28.699 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89748.log: /var/log/ceph/cluster1-client.mirror.149157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89151.log 2024-09-18T00:42:28.699 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84436.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146241.log 2024-09-18T00:42:28.700 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89151.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86038.log 2024-09-18T00:42:28.700 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59623.log 2024-09-18T00:42:28.700 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146241.log: /var/log/ceph/cluster1-client.mirror.86038.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64635.log 2024-09-18T00:42:28.700 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.59623.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85974.log 2024-09-18T00:42:28.700 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114377.log 2024-09-18T00:42:28.700 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137451.log 2024-09-18T00:42:28.700 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114377.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137007.log.gz 2024-09-18T00:42:28.700 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68352.log 2024-09-18T00:42:28.700 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137451.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99156.log 2024-09-18T00:42:28.700 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58955.log 2024-09-18T00:42:28.700 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61511.log 2024-09-18T00:42:28.700 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.58955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116869.log 2024-09-18T00:42:28.700 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.61511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109228.log 2024-09-18T00:42:28.701 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.116869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106365.log 2024-09-18T00:42:28.701 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71136.log 2024-09-18T00:42:28.701 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.106365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103425.log 2024-09-18T00:42:28.701 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88157.log 2024-09-18T00:42:28.701 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.103425.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68143.log 2024-09-18T00:42:28.701 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125853.log.gz 2024-09-18T00:42:28.701 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137280.log 2024-09-18T00:42:28.701 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146534.log 2024-09-18T00:42:28.701 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137280.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62534.log 2024-09-18T00:42:28.701 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146534.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72369.log.gz 2024-09-18T00:42:28.701 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.72321.log.gz 2024-09-18T00:42:28.701 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50870.log 2024-09-18T00:42:28.702 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148313.log 2024-09-18T00:42:28.702 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56262.log 2024-09-18T00:42:28.702 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94701.log 2024-09-18T00:42:28.702 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.146534.log.gz 2024-09-18T00:42:28.702 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.56262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114082.log 2024-09-18T00:42:28.702 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.94701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108048.log 2024-09-18T00:42:28.702 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114082.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.148313.log.gz 2024-09-18T00:42:28.702 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111141.log 2024-09-18T00:42:28.702 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.108048.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85739.log 2024-09-18T00:42:28.702 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111141.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41184.log 2024-09-18T00:42:28.702 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90137.log 2024-09-18T00:42:28.702 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126080.log.gz 2024-09-18T00:42:28.702 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41184.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92756.log 2024-09-18T00:42:28.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38058.log 2024-09-18T00:42:28.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.92756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149323.log 2024-09-18T00:42:28.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38058.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138372.log 2024-09-18T00:42:28.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95617.log 2024-09-18T00:42:28.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114375.log 2024-09-18T00:42:28.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71910.log 2024-09-18T00:42:28.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96845.log 2024-09-18T00:42:28.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71910.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46403.log 2024-09-18T00:42:28.703 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135136.log.gz 2024-09-18T00:42:28.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96845.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63625.log 2024-09-18T00:42:28.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46403.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98275.log 0.0% 2024-09-18T00:42:28.703 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.46403.log.gz 2024-09-18T00:42:28.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63625.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65935.log 2024-09-18T00:42:28.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51711.log 2024-09-18T00:42:28.704 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65935.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89003.log 2024-09-18T00:42:28.704 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51711.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117278.log 2024-09-18T00:42:28.704 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50473.log 2024-09-18T00:42:28.704 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.117278.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45695.log 2024-09-18T00:42:28.704 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.117278.log.gz 2024-09-18T00:42:28.704 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69049.log 2024-09-18T00:42:28.704 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45695.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58324.log 2024-09-18T00:42:28.704 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.69049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71973.log 2024-09-18T00:42:28.704 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.58324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34189.log 2024-09-18T00:42:28.704 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71973.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.87825.log.gz 2024-09-18T00:42:28.704 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40549.log 2024-09-18T00:42:28.704 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.34189.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113958.log 2024-09-18T00:42:28.705 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34189.log.gz 2024-09-18T00:42:28.705 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40549.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54440.log 2024-09-18T00:42:28.705 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113958.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50343.log 2024-09-18T00:42:28.705 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101006.log.gz 2024-09-18T00:42:28.705 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66649.log 2024-09-18T00:42:28.705 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130807.log 2024-09-18T00:42:28.705 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66649.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135935.log 2024-09-18T00:42:28.705 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130807.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58251.log 2024-09-18T00:42:28.705 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135935.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.69960.log 2024-09-18T00:42:28.705 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.58251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126041.log 2024-09-18T00:42:28.705 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.69960.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113535.log 2024-09-18T00:42:28.705 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55964.log 2024-09-18T00:42:28.705 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126041.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77956.log.gz 2024-09-18T00:42:28.706 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151304.log 2024-09-18T00:42:28.706 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55964.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42112.log 2024-09-18T00:42:28.706 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36532.log 2024-09-18T00:42:28.706 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90136.log 2024-09-18T00:42:28.706 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127536.log 2024-09-18T00:42:28.706 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.99956.log.gz 2024-09-18T00:42:28.706 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94769.log 2024-09-18T00:42:28.706 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71710.log 2024-09-18T00:42:28.706 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.94769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56459.log 2024-09-18T00:42:28.706 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.71710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77765.log 2024-09-18T00:42:28.706 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56459.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.151890.log 2024-09-18T00:42:28.706 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77765.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127219.log 2024-09-18T00:42:28.707 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.151890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98234.log 2024-09-18T00:42:28.707 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127219.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105202.log 2024-09-18T00:42:28.707 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98234.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97314.log 2024-09-18T00:42:28.707 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45451.log 2024-09-18T00:42:28.707 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.105202.log: /var/log/ceph/cluster1-client.mirror.97314.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44607.log.gz 2024-09-18T00:42:28.707 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153131.log 2024-09-18T00:42:28.707 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45451.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71099.log 2024-09-18T00:42:28.707 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.153131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87477.log 2024-09-18T00:42:28.707 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70603.log 2024-09-18T00:42:28.707 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122343.log 2024-09-18T00:42:28.707 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35317.log 2024-09-18T00:42:28.707 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148482.log 2024-09-18T00:42:28.707 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.35317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78462.log 2024-09-18T00:42:28.707 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster1-client.mirror.148482.log: -5 --verbose -- /var/log/ceph/cluster2-client.admin.35385.log 2024-09-18T00:42:28.708 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93857.log 2024-09-18T00:42:28.708 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.35385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119153.log 2024-09-18T00:42:28.708 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35385.log.gz 2024-09-18T00:42:28.708 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.93857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63145.log 2024-09-18T00:42:28.708 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149874.log 2024-09-18T00:42:28.708 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63145.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51187.log 2024-09-18T00:42:28.708 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129921.log 2024-09-18T00:42:28.708 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113545.log 2024-09-18T00:42:28.708 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114587.log 2024-09-18T00:42:28.708 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147978.log 2024-09-18T00:42:28.708 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66105.log 2024-09-18T00:42:28.708 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94400.log 2024-09-18T00:42:28.709 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66105.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.153131.log.gz 2024-09-18T00:42:28.709 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126819.log 2024-09-18T00:42:28.709 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.94400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61919.log 2024-09-18T00:42:28.709 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster1-client.mirror.126819.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90433.log 2024-09-18T00:42:28.709 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.61919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135011.log 2024-09-18T00:42:28.709 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152039.log 2024-09-18T00:42:28.709 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135011.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40010.log 2024-09-18T00:42:28.709 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111431.log 2024-09-18T00:42:28.709 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.81727.log 2024-09-18T00:42:28.709 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40640.log 2024-09-18T00:42:28.709 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97922.log.gz 2024-09-18T00:42:28.709 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.81727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78080.log 2024-09-18T00:42:28.709 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119216.log 2024-09-18T00:42:28.710 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78080.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46936.log 2024-09-18T00:42:28.710 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119216.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.127535.log.gz 2024-09-18T00:42:28.710 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143405.log 2024-09-18T00:42:28.710 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.46936.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48788.log 2024-09-18T00:42:28.710 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.152969.log 2024-09-18T00:42:28.710 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143405.log: /var/log/ceph/cluster1-client.mirror.48788.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109752.log 2024-09-18T00:42:28.710 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.152969.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67197.log 2024-09-18T00:42:28.710 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.148423.log.gz 2024-09-18T00:42:28.710 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79595.log 2024-09-18T00:42:28.710 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67197.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100462.log 2024-09-18T00:42:28.710 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79595.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76570.log 2024-09-18T00:42:28.710 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.79595.log.gz 2024-09-18T00:42:28.710 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100462.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.105628.log.gz 2024-09-18T00:42:28.711 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50785.log 2024-09-18T00:42:28.711 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76570.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104425.log 2024-09-18T00:42:28.711 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.152969.log.gz 2024-09-18T00:42:28.711 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147473.log 2024-09-18T00:42:28.711 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.104425.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55513.log 2024-09-18T00:42:28.711 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147473.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99305.log 2024-09-18T00:42:28.711 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88815.log 2024-09-18T00:42:28.711 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76525.log 2024-09-18T00:42:28.711 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59069.log 2024-09-18T00:42:28.711 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76525.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110972.log 2024-09-18T00:42:28.711 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.59069.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62910.log 2024-09-18T00:42:28.711 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74623.log 2024-09-18T00:42:28.711 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110972.log: /var/log/ceph/cluster1-client.mirror.62910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113326.log 2024-09-18T00:42:28.712 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74623.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95860.log 2024-09-18T00:42:28.712 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113326.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62196.log.gz 2024-09-18T00:42:28.712 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118691.log 2024-09-18T00:42:28.712 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95860.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46305.log 2024-09-18T00:42:28.712 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster1-client.admin.118691.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133754.log 2024-09-18T00:42:28.712 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146217.log 2024-09-18T00:42:28.712 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142203.log 2024-09-18T00:42:28.712 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146217.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64089.log 2024-09-18T00:42:28.712 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142203.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65035.log 2024-09-18T00:42:28.712 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68227.log 2024-09-18T00:42:28.712 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65035.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86957.log 2024-09-18T00:42:28.712 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89506.log 2024-09-18T00:42:28.712 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76339.log 2024-09-18T00:42:28.713 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138290.log 2024-09-18T00:42:28.713 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90811.log 2024-09-18T00:42:28.713 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.mirror.138290.log: /var/log/ceph/cluster2-client.mirror.94567.log 2024-09-18T00:42:28.713 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55449.log 2024-09-18T00:42:28.713 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41273.log 2024-09-18T00:42:28.713 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.94567.log: /var/log/ceph/cluster1-client.mirror.55449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101466.log 2024-09-18T00:42:28.713 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55659.log 2024-09-18T00:42:28.713 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101466.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101298.log 2024-09-18T00:42:28.713 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99493.log 2024-09-18T00:42:28.713 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.119610.log.gz 2024-09-18T00:42:28.713 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94914.log 2024-09-18T00:42:28.713 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147956.log 2024-09-18T00:42:28.713 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.94914.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133687.log 2024-09-18T00:42:28.714 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147956.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133542.log 2024-09-18T00:42:28.714 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95536.log 2024-09-18T00:42:28.714 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150842.log 2024-09-18T00:42:28.714 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95536.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47373.log 2024-09-18T00:42:28.714 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46495.log 2024-09-18T00:42:28.714 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.147956.log.gz 2024-09-18T00:42:28.714 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54104.log.gz 2024-09-18T00:42:28.714 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.47373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126461.log 2024-09-18T00:42:28.714 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46495.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44002.log 2024-09-18T00:42:28.714 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46495.log.gz 2024-09-18T00:42:28.714 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72146.log 2024-09-18T00:42:28.714 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144934.log 2024-09-18T00:42:28.714 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72146.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110596.log 2024-09-18T00:42:28.715 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148213.log 2024-09-18T00:42:28.715 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110596.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96249.log 2024-09-18T00:42:28.715 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56341.log 2024-09-18T00:42:28.715 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148213.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123770.log.gz 2024-09-18T00:42:28.715 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96249.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121964.log 2024-09-18T00:42:28.715 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149263.log 2024-09-18T00:42:28.715 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121964.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40934.log 2024-09-18T00:42:28.715 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148213.log.gz 2024-09-18T00:42:28.715 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129176.log 2024-09-18T00:42:28.715 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43800.log.gz 2024-09-18T00:42:28.715 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85347.log 2024-09-18T00:42:28.715 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129176.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150165.log 2024-09-18T00:42:28.715 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134634.log 2024-09-18T00:42:28.716 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62994.log 2024-09-18T00:42:28.716 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134634.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118456.log 2024-09-18T00:42:28.716 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62994.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66564.log 2024-09-18T00:42:28.716 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.118456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71558.log 2024-09-18T00:42:28.716 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39213.log 2024-09-18T00:42:28.716 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.71558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60411.log 2024-09-18T00:42:28.716 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39213.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82908.log 2024-09-18T00:42:28.716 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60411.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127635.log 2024-09-18T00:42:28.716 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.82908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44369.log 2024-09-18T00:42:28.716 INFO:teuthology.orchestra.run.smithi192.stderr: 66.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58593.log.gz 2024-09-18T00:42:28.716 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115158.log 2024-09-18T00:42:28.716 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129103.log 2024-09-18T00:42:28.717 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115158.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.115158.log.gz 2024-09-18T00:42:28.717 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82736.log 2024-09-18T00:42:28.717 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.129103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147582.log 2024-09-18T00:42:28.717 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.82736.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129948.log 2024-09-18T00:42:28.717 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64740.log 2024-09-18T00:42:28.717 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129948.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94494.log 2024-09-18T00:42:28.717 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87955.log 2024-09-18T00:42:28.717 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.94494.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56188.log 2024-09-18T00:42:28.717 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117430.log 2024-09-18T00:42:28.717 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38663.log.gz 2024-09-18T00:42:28.717 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56188.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.56188.log.gz 2024-09-18T00:42:28.717 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81355.log 2024-09-18T00:42:28.717 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.117430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82866.log 2024-09-18T00:42:28.717 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41482.log 2024-09-18T00:42:28.718 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.82866.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94225.log 2024-09-18T00:42:28.718 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146112.log 2024-09-18T00:42:28.718 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.94225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87617.log 2024-09-18T00:42:28.718 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146112.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81773.log 2024-09-18T00:42:28.718 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123812.log 2024-09-18T00:42:28.718 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.81773.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146008.log 2024-09-18T00:42:28.718 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123812.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.56354.log.gz 2024-09-18T00:42:28.718 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46482.log 2024-09-18T00:42:28.718 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.146112.log.gz 2024-09-18T00:42:28.718 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81133.log 2024-09-18T00:42:28.718 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146008.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.56674.log.gz 2024-09-18T00:42:28.718 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46482.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46482.log.gz 2024-09-18T00:42:28.718 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34273.log 2024-09-18T00:42:28.719 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81133.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139269.log 2024-09-18T00:42:28.719 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79546.log 2024-09-18T00:42:28.719 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.34273.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34273.log.gz 2024-09-18T00:42:28.719 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74690.log 2024-09-18T00:42:28.719 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79546.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41273.log.gz 2024-09-18T00:42:28.719 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109965.log 2024-09-18T00:42:28.719 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119259.log 2024-09-18T00:42:28.719 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74690.log: /var/log/ceph/cluster1-client.mirror.109965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122009.log 2024-09-18T00:42:28.719 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.119259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77366.log 2024-09-18T00:42:28.719 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148064.log 2024-09-18T00:42:28.719 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121966.log 2024-09-18T00:42:28.719 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153313.log 2024-09-18T00:42:28.719 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121966.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61536.log 2024-09-18T00:42:28.720 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.153313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139707.log 2024-09-18T00:42:28.720 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.61536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36381.log 2024-09-18T00:42:28.720 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139707.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117886.log 2024-09-18T00:42:28.720 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70241.log 2024-09-18T00:42:28.720 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36381.log: 91.0% -- replaced with /var/log/ceph/cluster1-client.admin.153313.log.gz 2024-09-18T00:42:28.720 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.117886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106406.log 2024-09-18T00:42:28.720 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68378.log 2024-09-18T00:42:28.720 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106406.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48332.log 2024-09-18T00:42:28.720 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68378.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78500.log 2024-09-18T00:42:28.720 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142799.log 2024-09-18T00:42:28.720 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.128134.log.gz 2024-09-18T00:42:28.720 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48332.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.128133.log.gz 2024-09-18T00:42:28.720 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78500.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80251.log 2024-09-18T00:42:28.721 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41812.log 2024-09-18T00:42:28.721 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.80251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77010.log 2024-09-18T00:42:28.721 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41812.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64028.log 2024-09-18T00:42:28.721 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57648.log 2024-09-18T00:42:28.721 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106690.log 2024-09-18T00:42:28.721 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44335.log 2024-09-18T00:42:28.721 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106690.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152043.log 2024-09-18T00:42:28.721 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44335.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75508.log.gz 2024-09-18T00:42:28.721 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44335.log.gz 2024-09-18T00:42:28.721 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138077.log 2024-09-18T00:42:28.721 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152043.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139633.log 2024-09-18T00:42:28.721 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138077.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103245.log 2024-09-18T00:42:28.721 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80821.log 2024-09-18T00:42:28.722 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.103245.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100846.log 2024-09-18T00:42:28.722 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55490.log 2024-09-18T00:42:28.722 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123536.log 2024-09-18T00:42:28.722 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55490.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55766.log 2024-09-18T00:42:28.722 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123536.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.151614.log 2024-09-18T00:42:28.722 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103467.log 2024-09-18T00:42:28.722 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.151614.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33994.log 2024-09-18T00:42:28.722 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103467.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.103467.log.gz 2024-09-18T00:42:28.722 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140773.log 2024-09-18T00:42:28.722 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33994.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104077.log 2024-09-18T00:42:28.722 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33994.log.gz 2024-09-18T00:42:28.722 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140773.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140773.log.gz 2024-09-18T00:42:28.722 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40600.log 2024-09-18T00:42:28.723 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.104077.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67068.log 2024-09-18T00:42:28.723 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40600.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.88269.log.gz 2024-09-18T00:42:28.723 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135556.log 2024-09-18T00:42:28.723 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79825.log 2024-09-18T00:42:28.723 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135556.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137089.log 2024-09-18T00:42:28.723 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40551.log/var/log/ceph/cluster1-client.admin.79825.log: 2024-09-18T00:42:28.723 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48178.log 2024-09-18T00:42:28.723 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40551.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35471.log 2024-09-18T00:42:28.723 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.89152.log.gz 2024-09-18T00:42:28.723 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142687.log 2024-09-18T00:42:28.723 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.35471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52130.log 2024-09-18T00:42:28.723 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.142687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135934.log 2024-09-18T00:42:28.723 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52130.log: /var/log/ceph/cluster1-client.mirror.135934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87953.log 2024-09-18T00:42:28.723 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102897.log.gz 2024-09-18T00:42:28.724 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81981.log 2024-09-18T00:42:28.724 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102896.log.gz 2024-09-18T00:42:28.724 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135559.log.gz 2024-09-18T00:42:28.724 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124987.log 2024-09-18T00:42:28.724 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.81981.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133481.log 2024-09-18T00:42:28.724 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92833.log 2024-09-18T00:42:28.724 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108180.log 2024-09-18T00:42:28.724 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88119.log 2024-09-18T00:42:28.724 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152254.log 2024-09-18T00:42:28.724 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88119.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64088.log 2024-09-18T00:42:28.724 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.152254.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107768.log 2024-09-18T00:42:28.724 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109350.log 2024-09-18T00:42:28.910 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148111.log 2024-09-18T00:42:28.910 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109350.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57249.log 2024-09-18T00:42:28.910 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81357.log 2024-09-18T00:42:28.910 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.57249.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149474.log 2024-09-18T00:42:28.910 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137175.log.gz 2024-09-18T00:42:28.911 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119612.log 2024-09-18T00:42:28.911 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148111.log.gz 2024-09-18T00:42:28.911 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149474.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136060.log 2024-09-18T00:42:28.911 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39550.log 2024-09-18T00:42:28.911 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136060.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41274.log 2024-09-18T00:42:28.911 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44577.log 2024-09-18T00:42:28.911 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35815.log 2024-09-18T00:42:28.911 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44577.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44577.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100882.log 2024-09-18T00:42:28.911 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:28.911 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cluster1-client.mirror.35815.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.91606.log 2024-09-18T00:42:28.911 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100882.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73971.log.gz 2024-09-18T00:42:28.911 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134234.log 2024-09-18T00:42:28.911 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52298.log 2024-09-18T00:42:28.912 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91606.log.gz 2024-09-18T00:42:28.912 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134234.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84900.log 2024-09-18T00:42:28.912 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134506.log 2024-09-18T00:42:28.912 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84900.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133077.log 2024-09-18T00:42:28.912 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45752.log 2024-09-18T00:42:28.912 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134339.log.gz 2024-09-18T00:42:28.912 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129538.log 2024-09-18T00:42:28.912 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95447.log 2024-09-18T00:42:28.912 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129538.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124232.log 2024-09-18T00:42:28.912 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.134300.log.gz 2024-09-18T00:42:28.912 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88666.log 2024-09-18T00:42:28.912 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145145.log 2024-09-18T00:42:28.913 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88666.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135895.log 2024-09-18T00:42:28.913 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124318.log 2024-09-18T00:42:28.913 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145145.log: /var/log/ceph/cluster1-client.mirror.135895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53642.log 2024-09-18T00:42:28.913 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124318.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51103.log 2024-09-18T00:42:28.913 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124862.log 2024-09-18T00:42:28.913 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86647.log 2024-09-18T00:42:28.913 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124862.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73514.log 2024-09-18T00:42:28.913 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149156.log 2024-09-18T00:42:28.913 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50701.log 2024-09-18T00:42:28.913 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149156.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34483.log 2024-09-18T00:42:28.913 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48742.log 2024-09-18T00:42:28.914 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.34483.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68709.log 2024-09-18T00:42:28.914 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34483.log.gz 2024-09-18T00:42:28.914 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.48742.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70489.log 2024-09-18T00:42:28.914 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.68709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42423.log 2024-09-18T00:42:28.914 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70489.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71272.log 2024-09-18T00:42:28.914 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100167.log 2024-09-18T00:42:28.914 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95795.log 2024-09-18T00:42:28.914 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100167.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42955.log 2024-09-18T00:42:28.914 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95795.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77620.log 2024-09-18T00:42:28.914 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1%/var/log/ceph/cluster1-client.mirror.42955.log: -- replaced with /var/log/ceph/cluster1-client.mirror.66648.log.gz 2024-09-18T00:42:28.914 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127440.log 2024-09-18T00:42:28.914 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77620.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41904.log 2024-09-18T00:42:28.914 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.66649.log.gz 2024-09-18T00:42:28.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103269.log 2024-09-18T00:42:28.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91433.log 2024-09-18T00:42:28.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103269.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.103269.log.gz 2024-09-18T00:42:28.915 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140316.log 2024-09-18T00:42:28.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97812.log 2024-09-18T00:42:28.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140316.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113327.log 2024-09-18T00:42:28.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97812.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122174.log 2024-09-18T00:42:28.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113327.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131346.log 2024-09-18T00:42:28.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51102.log 2024-09-18T00:42:28.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60825.log 2024-09-18T00:42:28.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51102.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114955.log 2024-09-18T00:42:28.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60825.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83185.log 2024-09-18T00:42:28.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76465.log 2024-09-18T00:42:28.916 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.83185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40639.log 2024-09-18T00:42:28.916 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54990.log 2024-09-18T00:42:28.916 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40639.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131089.log 2024-09-18T00:42:28.916 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54990.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98360.log 2024-09-18T00:42:28.916 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127021.log 2024-09-18T00:42:28.916 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59469.log 2024-09-18T00:42:28.916 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.127021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136190.log 2024-09-18T00:42:28.916 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59469.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.81955.log 2024-09-18T00:42:28.916 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136439.log 2024-09-18T00:42:28.916 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.81955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69651.log 2024-09-18T00:42:28.916 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136439.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121076.log 2024-09-18T00:42:28.916 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49025.log.gz 2024-09-18T00:42:28.917 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.49004.log.gz 2024-09-18T00:42:28.917 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94286.log 2024-09-18T00:42:28.917 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121076.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132838.log 2024-09-18T00:42:28.917 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.94286.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123688.log 2024-09-18T00:42:28.917 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.132838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101592.log 2024-09-18T00:42:28.917 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123688.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50786.log 2024-09-18T00:42:28.917 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148944.log 2024-09-18T00:42:28.917 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58438.log 2024-09-18T00:42:28.917 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77473.log 2024-09-18T00:42:28.917 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.58438.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46389.log 2024-09-18T00:42:28.917 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77473.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77365.log 2024-09-18T00:42:28.917 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46389.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69400.log 2024-09-18T00:42:28.917 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46389.log.gz 2024-09-18T00:42:28.918 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78080.log.gz 2024-09-18T00:42:28.918 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40644.log 2024-09-18T00:42:28.918 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70530.log 2024-09-18T00:42:28.918 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54650.log 2024-09-18T00:42:28.918 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70530.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57317.log 2024-09-18T00:42:28.918 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142165.log 2024-09-18T00:42:28.918 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54650.log: /var/log/ceph/cluster2-client.admin.57317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122890.log 2024-09-18T00:42:28.918 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126419.log 2024-09-18T00:42:28.918 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72567.log 2024-09-18T00:42:28.918 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42863.log 2024-09-18T00:42:28.918 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114584.log 2024-09-18T00:42:28.918 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42863.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35999.log 2024-09-18T00:42:28.918 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119001.log 2024-09-18T00:42:28.919 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.35999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57269.log 2024-09-18T00:42:28.919 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65830.log 2024-09-18T00:42:28.919 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137279.log 2024-09-18T00:42:28.919 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55964.log.gz 2024-09-18T00:42:28.919 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65830.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102139.log 2024-09-18T00:42:28.919 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137279.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104376.log 2024-09-18T00:42:28.919 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42862.log 2024-09-18T00:42:28.919 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.104376.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81867.log 2024-09-18T00:42:28.919 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42862.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48789.log 2024-09-18T00:42:28.919 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.81867.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112442.log 2024-09-18T00:42:28.919 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48789.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129036.log 2024-09-18T00:42:28.919 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72146.log.gz 2024-09-18T00:42:28.919 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89380.log 2024-09-18T00:42:28.920 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129036.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115599.log 2024-09-18T00:42:28.920 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117612.log 2024-09-18T00:42:28.920 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115599.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82525.log 2024-09-18T00:42:28.920 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.117612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77054.log 2024-09-18T00:42:28.920 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.82525.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46002.log 2024-09-18T00:42:28.920 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77054.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89549.log 2024-09-18T00:42:28.920 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117309.log 2024-09-18T00:42:28.920 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89549.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117004.log 2024-09-18T00:42:28.920 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.117309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125009.log 2024-09-18T00:42:28.920 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.117309.log.gz 2024-09-18T00:42:28.920 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.117004.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127729.log 2024-09-18T00:42:28.920 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99201.log 2024-09-18T00:42:28.920 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.127729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118955.log 2024-09-18T00:42:28.920 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99201.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68988.log 2024-09-18T00:42:28.921 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.118955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76191.log 2024-09-18T00:42:28.921 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.68988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48483.log 2024-09-18T00:42:28.921 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113763.log 2024-09-18T00:42:28.921 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115314.log 2024-09-18T00:42:28.921 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44173.log 2024-09-18T00:42:28.921 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107310.log 2024-09-18T00:42:28.921 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78795.log.gz 2024-09-18T00:42:28.921 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60564.log 2024-09-18T00:42:28.921 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36139.log 2024-09-18T00:42:28.921 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127323.log 2024-09-18T00:42:28.921 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.36139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66815.log 2024-09-18T00:42:28.921 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103757.log 2024-09-18T00:42:28.921 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40432.log 2024-09-18T00:42:28.922 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67737.log 2024-09-18T00:42:28.922 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149743.log 2024-09-18T00:42:28.922 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.67737.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.67737.log.gz 2024-09-18T00:42:28.922 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107831.log 2024-09-18T00:42:28.922 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44829.log 2024-09-18T00:42:28.922 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107831.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37701.log.gz 2024-09-18T00:42:28.922 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85975.log 2024-09-18T00:42:28.922 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44829.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74836.log 2024-09-18T00:42:28.922 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98736.log 2024-09-18T00:42:28.922 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74836.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103326.log 2024-09-18T00:42:28.922 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98736.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91813.log 2024-09-18T00:42:28.922 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.103326.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105476.log 2024-09-18T00:42:28.923 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65349.log 2024-09-18T00:42:28.923 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.105476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121503.log 2024-09-18T00:42:28.923 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105476.log.gz 2024-09-18T00:42:28.923 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65349.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75362.log 2024-09-18T00:42:28.923 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58666.log 2024-09-18T00:42:28.923 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122051.log 2024-09-18T00:42:28.923 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.58666.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78585.log 2024-09-18T00:42:28.923 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147053.log 2024-09-18T00:42:28.923 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78585.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145983.log 2024-09-18T00:42:28.923 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147053.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79155.log.gz 2024-09-18T00:42:28.923 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79726.log 2024-09-18T00:42:28.923 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145983.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74161.log 2024-09-18T00:42:28.923 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79726.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94111.log 2024-09-18T00:42:28.924 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74161.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65766.log 2024-09-18T00:42:28.924 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.94111.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138774.log 2024-09-18T00:42:28.924 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40813.log 2024-09-18T00:42:28.924 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87553.log 2024-09-18T00:42:28.924 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77265.log 2024-09-18T00:42:28.924 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87553.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66227.log 2024-09-18T00:42:28.924 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100462.log.gz 2024-09-18T00:42:28.924 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77265.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87660.log 2024-09-18T00:42:28.924 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66227.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.151898.log 2024-09-18T00:42:28.924 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87660.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102477.log 2024-09-18T00:42:28.924 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.151898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65536.log 2024-09-18T00:42:28.924 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54736.log 2024-09-18T00:42:28.925 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65536.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128371.log 2024-09-18T00:42:28.925 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54736.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130185.log 2024-09-18T00:42:28.925 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143257.log 2024-09-18T00:42:28.925 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137280.log.gz 2024-09-18T00:42:28.925 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141811.log 2024-09-18T00:42:28.925 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55070.log 2024-09-18T00:42:28.925 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142949.log 2024-09-18T00:42:28.925 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55070.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93612.log 2024-09-18T00:42:28.925 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134360.log 2024-09-18T00:42:28.925 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93612.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.93612.log.gz 2024-09-18T00:42:28.925 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90323.log 2024-09-18T00:42:28.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60518.log 2024-09-18T00:42:28.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135809.log 2024-09-18T00:42:28.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.60518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117471.log 2024-09-18T00:42:28.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135809.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68144.log 2024-09-18T00:42:28.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.117471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37081.log 2024-09-18T00:42:28.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56818.log 2024-09-18T00:42:28.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53057.log 2024-09-18T00:42:28.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59991.log 2024-09-18T00:42:28.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53728.log 2024-09-18T00:42:28.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59991.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149575.log 2024-09-18T00:42:28.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113741.log 2024-09-18T00:42:28.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114168.log 2024-09-18T00:42:28.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105950.log 2024-09-18T00:42:28.927 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123626.log 2024-09-18T00:42:28.927 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.105950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146383.log 2024-09-18T00:42:28.927 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135894.log 2024-09-18T00:42:28.927 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101912.log 2024-09-18T00:42:28.927 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71908.log 2024-09-18T00:42:28.927 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75825.log 2024-09-18T00:42:28.927 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153199.log 2024-09-18T00:42:28.927 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75825.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53012.log 2024-09-18T00:42:28.927 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.153199.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93599.log 2024-09-18T00:42:28.927 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77811.log 2024-09-18T00:42:28.927 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93599.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.93599.log.gz 2024-09-18T00:42:28.927 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139706.log 2024-09-18T00:42:28.927 INFO:teuthology.orchestra.run.smithi192.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.admin.153199.log.gz 2024-09-18T00:42:28.927 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96846.log 2024-09-18T00:42:28.928 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139706.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39214.log 2024-09-18T00:42:28.928 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96846.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45943.log 2024-09-18T00:42:28.928 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149053.log 2024-09-18T00:42:28.928 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.45943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90223.log 2024-09-18T00:42:28.928 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149053.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93473.log 2024-09-18T00:42:28.928 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65286.log 2024-09-18T00:42:28.928 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93473.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98820.log 2024-09-18T00:42:28.928 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.89748.log.gz 2024-09-18T00:42:28.928 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65286.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147054.log 2024-09-18T00:42:28.928 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146718.log 2024-09-18T00:42:28.928 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57345.log 2024-09-18T00:42:28.928 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129559.log 2024-09-18T00:42:28.929 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57345.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60939.log 2024-09-18T00:42:28.929 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.129559.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146718.log.gz 2024-09-18T00:42:28.929 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137509.log 2024-09-18T00:42:28.929 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72857.log.gz 2024-09-18T00:42:28.929 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60939.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43586.log 2024-09-18T00:42:28.929 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91923.log 2024-09-18T00:42:28.929 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43586.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102223.log 2024-09-18T00:42:28.929 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91923.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36248.log 2024-09-18T00:42:28.929 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51102.log.gz/var/log/ceph/cluster1-client.mirror.102223.log: 2024-09-18T00:42:28.929 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67321.log 2024-09-18T00:42:28.929 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36248.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142467.log 2024-09-18T00:42:28.929 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76039.log 2024-09-18T00:42:28.930 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142467.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38449.log 2024-09-18T00:42:28.930 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64883.log 2024-09-18T00:42:28.930 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82343.log 2024-09-18T00:42:28.930 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124509.log 2024-09-18T00:42:28.930 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.82343.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69896.log 2024-09-18T00:42:28.930 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109228.log.gz 2024-09-18T00:42:28.930 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134569.log 2024-09-18T00:42:28.930 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69896.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.69896.log.gz 2024-09-18T00:42:28.930 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83366.log 2024-09-18T00:42:28.930 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134569.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109350.log.gz 2024-09-18T00:42:28.930 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128469.log 2024-09-18T00:42:28.930 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146066.log 2024-09-18T00:42:28.930 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128469.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54187.log 2024-09-18T00:42:28.930 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.152855.log 2024-09-18T00:42:28.931 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.152741.log 2024-09-18T00:42:28.931 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.152855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61991.log 2024-09-18T00:42:28.931 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.152741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129351.log 2024-09-18T00:42:28.931 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.61991.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131262.log 2024-09-18T00:42:28.931 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129351.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.107526.log.gz 2024-09-18T00:42:28.931 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111224.log 2024-09-18T00:42:28.931 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.152855.log.gz 2024-09-18T00:42:28.931 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87554.log 2024-09-18T00:42:28.931 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59839.log 2024-09-18T00:42:28.931 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87554.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109326.log 2024-09-18T00:42:28.931 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5%/var/log/ceph/cluster1-client.mirror.59839.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129290.log 2024-09-18T00:42:28.931 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102013.log 2024-09-18T00:42:28.931 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.152741.log.gz 2024-09-18T00:42:28.932 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111853.log 2024-09-18T00:42:28.932 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102351.log.gz/var/log/ceph/cluster1-client.mirror.102013.log: 2024-09-18T00:42:28.932 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126970.log 2024-09-18T00:42:28.932 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100653.log 2024-09-18T00:42:28.932 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46802.log 2024-09-18T00:42:28.932 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102770.log 2024-09-18T00:42:28.932 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.46802.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69093.log 2024-09-18T00:42:28.932 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83520.log 2024-09-18T00:42:28.932 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69093.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140314.log 2024-09-18T00:42:28.932 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83520.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33723.log 2024-09-18T00:42:28.932 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42283.log 2024-09-18T00:42:28.932 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33723.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33723.log.gz 2024-09-18T00:42:28.932 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124510.log 2024-09-18T00:42:28.933 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88499.log 2024-09-18T00:42:28.933 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124510.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.84900.log.gz 2024-09-18T00:42:28.933 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124381.log 2024-09-18T00:42:28.933 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109672.log 2024-09-18T00:42:28.933 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124381.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150326.log 2024-09-18T00:42:28.933 INFO:teuthology.orchestra.run.smithi192.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36623.log.gz 2024-09-18T00:42:28.933 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133668.log 2024-09-18T00:42:28.933 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150326.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.150326.log.gz 2024-09-18T00:42:28.933 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74314.log 2024-09-18T00:42:28.933 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93883.log 2024-09-18T00:42:28.933 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74314.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33957.log 2024-09-18T00:42:28.933 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.93883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49212.log 2024-09-18T00:42:28.933 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33957.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33957.log.gz 2024-09-18T00:42:28.934 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90091.log 2024-09-18T00:42:28.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49212.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59049.log 2024-09-18T00:42:28.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123333.log 2024-09-18T00:42:28.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85893.log 2024-09-18T00:42:28.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78248.log 2024-09-18T00:42:28.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85893.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78524.log 2024-09-18T00:42:28.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57652.log 2024-09-18T00:42:28.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78524.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116225.log 2024-09-18T00:42:28.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100547.log 2024-09-18T00:42:28.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134987.log 2024-09-18T00:42:28.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87397.log 2024-09-18T00:42:28.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134987.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76570.log.gz 2024-09-18T00:42:28.934 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50745.log 2024-09-18T00:42:28.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87397.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71820.log 2024-09-18T00:42:28.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50745.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111602.log 2024-09-18T00:42:28.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151333.log 2024-09-18T00:42:28.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76526.log 2024-09-18T00:42:28.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54991.log 2024-09-18T00:42:28.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76466.log 2024-09-18T00:42:28.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54991.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110302.log.gz 2024-09-18T00:42:28.935 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42113.log 2024-09-18T00:42:28.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76466.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125492.log 2024-09-18T00:42:28.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113072.log 2024-09-18T00:42:28.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100796.log 2024-09-18T00:42:28.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115465.log 2024-09-18T00:42:28.936 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108743.log 2024-09-18T00:42:28.936 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.115465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66167.log 2024-09-18T00:42:28.936 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40643.log 2024-09-18T00:42:28.936 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66167.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101511.log 2024-09-18T00:42:28.936 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.35629.log 2024-09-18T00:42:28.936 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115369.log 2024-09-18T00:42:28.936 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.0.35629.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65264.log 2024-09-18T00:42:28.936 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137574.log 2024-09-18T00:42:28.936 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65264.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116101.log 2024-09-18T00:42:28.936 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146843.log 2024-09-18T00:42:28.936 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52130.log.gz 2024-09-18T00:42:28.936 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136708.log 2024-09-18T00:42:28.936 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146843.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141697.log 2024-09-18T00:42:28.937 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58666.log.gz 2024-09-18T00:42:28.937 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136708.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146743.log 2024-09-18T00:42:28.937 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141697.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125071.log 2024-09-18T00:42:28.937 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146743.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45948.log 2024-09-18T00:42:28.937 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125071.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60105.log 2024-09-18T00:42:28.937 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.45948.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72099.log 2024-09-18T00:42:28.937 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46007.log 2024-09-18T00:42:28.937 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.146743.log.gz 2024-09-18T00:42:28.937 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105411.log 2024-09-18T00:42:28.937 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110972.log.gz 2024-09-18T00:42:28.937 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46007.log: gzip 80.1% -5 --verbose -- replaced with /var/log/ceph/cluster1-client.mirror.83520.log.gz -- 2024-09-18T00:42:28.937 INFO:teuthology.orchestra.run.smithi192.stderr: /var/log/ceph/cluster1-client.mirror.147162.log 2024-09-18T00:42:28.937 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.105411.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105411.log.gz 2024-09-18T00:42:28.937 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124274.log 2024-09-18T00:42:28.938 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147162.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34399.log 2024-09-18T00:42:28.938 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40429.log 2024-09-18T00:42:28.938 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.34399.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34399.log.gz 2024-09-18T00:42:28.938 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.147162.log.gz 2024-09-18T00:42:28.938 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102266.log 2024-09-18T00:42:28.938 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40429.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61991.log.gz 2024-09-18T00:42:28.938 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.151894.log 2024-09-18T00:42:28.938 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53961.log 2024-09-18T00:42:28.938 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.151894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102098.log 2024-09-18T00:42:28.938 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40221.log 2024-09-18T00:42:28.938 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41694.log 2024-09-18T00:42:28.938 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100376.log 2024-09-18T00:42:28.938 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41694.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90973.log 2024-09-18T00:42:28.939 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89379.log 2024-09-18T00:42:28.939 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107348.log 2024-09-18T00:42:28.939 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113911.log 2024-09-18T00:42:28.939 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107348.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.91813.log.gz 2024-09-18T00:42:28.939 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153339.log 2024-09-18T00:42:28.939 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111562.log 2024-09-18T00:42:28.939 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.153339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52382.log 2024-09-18T00:42:28.939 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.153339.log.gz 2024-09-18T00:42:28.939 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142335.log 2024-09-18T00:42:28.939 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138477.log 2024-09-18T00:42:28.939 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.108048.log.gz 2024-09-18T00:42:28.939 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37145.log 2024-09-18T00:42:28.940 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51250.log 2024-09-18T00:42:28.940 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37145.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152351.log 2024-09-18T00:42:28.940 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51250.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121546.log 2024-09-18T00:42:28.940 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.80179.log.gz 2024-09-18T00:42:28.940 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.152351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126374.log 2024-09-18T00:42:28.940 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138287.log 2024-09-18T00:42:28.940 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126374.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81688.log 2024-09-18T00:42:28.940 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138287.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35411.log 2024-09-18T00:42:28.940 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81688.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81688.log.gz 2024-09-18T00:42:28.940 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153628.log 2024-09-18T00:42:28.940 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.35411.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35411.log.gz 2024-09-18T00:42:28.940 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65959.log 2024-09-18T00:42:28.940 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.153628.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62323.log 2024-09-18T00:42:28.941 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65959.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.153628.log.gz 2024-09-18T00:42:28.941 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112902.log.gz 2024-09-18T00:42:28.941 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108493.log 2024-09-18T00:42:28.941 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40091.log 2024-09-18T00:42:28.941 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131795.log 2024-09-18T00:42:28.941 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.141013.log.gz 2024-09-18T00:42:28.941 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101676.log 2024-09-18T00:42:28.941 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131795.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50954.log 2024-09-18T00:42:28.941 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90768.log 2024-09-18T00:42:28.941 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66607.log 2024-09-18T00:42:28.941 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74121.log 2024-09-18T00:42:28.941 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66607.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56251.log 2024-09-18T00:42:28.942 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119307.log 2024-09-18T00:42:28.942 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56251.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.56251.log.gz 2024-09-18T00:42:28.942 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60736.log 2024-09-18T00:42:28.942 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137031.log 2024-09-18T00:42:28.942 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60736.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75825.log.gz 2024-09-18T00:42:28.942 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71606.log 2024-09-18T00:42:28.942 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95287.log 2024-09-18T00:42:28.942 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71606.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.104077.log.gz 2024-09-18T00:42:28.942 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58145.log 2024-09-18T00:42:28.942 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146172.log 2024-09-18T00:42:28.942 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.58145.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112063.log 2024-09-18T00:42:28.942 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.58145.log.gz 2024-09-18T00:42:28.942 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114291.log 2024-09-18T00:42:28.943 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112063.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86185.log 2024-09-18T00:42:28.943 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78626.log 2024-09-18T00:42:28.943 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111435.log 2024-09-18T00:42:28.943 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149993.log 2024-09-18T00:42:28.943 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3%/var/log/ceph/cluster1-client.mirror.111435.log: -- replaced with /var/log/ceph/cluster1-client.admin.56341.log.gz 2024-09-18T00:42:28.943 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59839.log.gz 2024-09-18T00:42:28.943 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62954.log 2024-09-18T00:42:28.943 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150143.log 2024-09-18T00:42:28.943 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62954.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115860.log 2024-09-18T00:42:28.943 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62726.log 2024-09-18T00:42:28.943 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.115860.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73929.log 2024-09-18T00:42:28.943 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62726.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94453.log 2024-09-18T00:42:28.943 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73929.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107679.log 2024-09-18T00:42:28.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.94453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77198.log 2024-09-18T00:42:28.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107679.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143711.log 2024-09-18T00:42:28.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77198.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53615.log 2024-09-18T00:42:28.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143711.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94266.log 2024-09-18T00:42:28.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149196.log 2024-09-18T00:42:28.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.94266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44006.log 2024-09-18T00:42:28.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149196.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125998.log 2024-09-18T00:42:28.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44006.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89652.log 2024-09-18T00:42:28.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125998.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146423.log 2024-09-18T00:42:28.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130343.log 2024-09-18T00:42:28.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79237.log 2024-09-18T00:42:28.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.130343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125702.log 2024-09-18T00:42:28.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79237.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100964.log 2024-09-18T00:42:28.945 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148796.log 2024-09-18T00:42:28.945 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100964.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57013.log 2024-09-18T00:42:28.945 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124022.log 2024-09-18T00:42:28.945 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41903.log 2024-09-18T00:42:28.945 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124022.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62534.log.gz 2024-09-18T00:42:28.945 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45571.log 2024-09-18T00:42:28.945 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39003.log 2024-09-18T00:42:28.945 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.45571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84612.log 2024-09-18T00:42:28.945 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149533.log 2024-09-18T00:42:28.945 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84612.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92407.log 2024-09-18T00:42:28.945 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149533.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70945.log 2024-09-18T00:42:28.945 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112822.log 2024-09-18T00:42:28.945 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64993.log 2024-09-18T00:42:28.946 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86562.log 2024-09-18T00:42:28.946 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40019.log.gz 2024-09-18T00:42:28.946 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64993.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59008.log 2024-09-18T00:42:28.946 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103296.log 2024-09-18T00:42:28.946 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.59008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48546.log 2024-09-18T00:42:28.946 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103296.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.103296.log.gz 2024-09-18T00:42:28.946 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147686.log 2024-09-18T00:42:28.946 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.57345.log.gz 2024-09-18T00:42:28.946 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44425.log 2024-09-18T00:42:28.946 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92266.log 2024-09-18T00:42:28.946 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44425.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44425.log.gz 2024-09-18T00:42:28.946 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141335.log 2024-09-18T00:42:28.946 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92266.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.57652.log.gz 2024-09-18T00:42:28.947 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95686.log 2024-09-18T00:42:28.947 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.141335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99830.log 2024-09-18T00:42:28.947 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66168.log 2024-09-18T00:42:28.947 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99830.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42324.log 2024-09-18T00:42:28.947 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153172.log 2024-09-18T00:42:28.947 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66019.log 2024-09-18T00:42:28.947 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.153172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146111.log 2024-09-18T00:42:28.947 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.153172.log.gz 2024-09-18T00:42:28.947 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66019.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.151793.log 2024-09-18T00:42:28.947 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146111.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108275.log 2024-09-18T00:42:28.947 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.151793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71850.log 2024-09-18T00:42:28.947 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.140785.log 2024-09-18T00:42:28.947 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.71850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40935.log 2024-09-18T00:42:28.948 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.140785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121245.log 2024-09-18T00:42:28.948 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40935.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151089.log 2024-09-18T00:42:28.948 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121245.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73858.log 2024-09-18T00:42:28.948 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.89298.log.gz 2024-09-18T00:42:28.948 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60667.log 2024-09-18T00:42:28.948 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127537.log 2024-09-18T00:42:28.948 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83158.log 2024-09-18T00:42:28.948 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36533.log 2024-09-18T00:42:28.948 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50269.log 2024-09-18T00:42:28.948 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67761.log 2024-09-18T00:42:28.948 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134425.log 2024-09-18T00:42:28.948 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67761.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104049.log 2024-09-18T00:42:28.948 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134425.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150360.log 2024-09-18T00:42:28.949 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138708.log 2024-09-18T00:42:28.949 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147835.log 2024-09-18T00:42:28.949 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.150360.log.gz 2024-09-18T00:42:28.949 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.138708.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.138708.log.gz 2024-09-18T00:42:28.949 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122848.log 2024-09-18T00:42:28.949 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111226.log 2024-09-18T00:42:28.949 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91433.log.gz 2024-09-18T00:42:28.949 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122848.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106848.log 2024-09-18T00:42:28.949 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.147835.log.gz 2024-09-18T00:42:28.949 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118477.log 2024-09-18T00:42:28.949 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.106848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92578.log 2024-09-18T00:42:28.949 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53222.log 2024-09-18T00:42:28.949 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92578.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111223.log 2024-09-18T00:42:28.950 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53222.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141770.log 2024-09-18T00:42:28.950 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96051.log 2024-09-18T00:42:28.950 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141770.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104067.log 2024-09-18T00:42:28.950 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96051.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138224.log 2024-09-18T00:42:28.950 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104067.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41062.log 2024-09-18T00:42:28.950 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94975.log 2024-09-18T00:42:28.950 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41062.log: gzip 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98234.log.gz -5 2024-09-18T00:42:28.950 INFO:teuthology.orchestra.run.smithi192.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.86812.log 2024-09-18T00:42:28.950 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.94975.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105383.log 2024-09-18T00:42:28.950 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86812.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54779.log 2024-09-18T00:42:28.950 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.105383.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60258.log 2024-09-18T00:42:28.950 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.105383.log.gz 2024-09-18T00:42:28.950 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54779.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125409.log 2024-09-18T00:42:28.950 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53517.log 2024-09-18T00:42:28.951 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141449.log 2024-09-18T00:42:28.951 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53517.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75172.log 2024-09-18T00:42:28.951 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.141449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117866.log 2024-09-18T00:42:28.951 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75172.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74121.log.gz 2024-09-18T00:42:28.951 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82809.log 2024-09-18T00:42:28.951 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.117866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133854.log 2024-09-18T00:42:28.951 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.82809.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40322.log 2024-09-18T00:42:28.951 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133854.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130148.log 2024-09-18T00:42:28.951 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58385.log 2024-09-18T00:42:28.951 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130148.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91715.log 2024-09-18T00:42:28.951 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0%/var/log/ceph/cluster1-client.admin.58385.log: -- replaced with /var/log/ceph/cluster1-client.mirror.135010.log.gz 2024-09-18T00:42:28.951 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137426.log 2024-09-18T00:42:28.951 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97287.log 2024-09-18T00:42:28.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70738.log 2024-09-18T00:42:28.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74037.log 2024-09-18T00:42:28.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.70738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114541.log 2024-09-18T00:42:28.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74037.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77430.log 2024-09-18T00:42:28.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114541.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.65959.log.gz 2024-09-18T00:42:28.952 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124780.log 2024-09-18T00:42:28.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135557.log 2024-09-18T00:42:28.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124780.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50165.log 2024-09-18T00:42:28.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90602.log 2024-09-18T00:42:28.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112068.log 2024-09-18T00:42:28.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114753.log 2024-09-18T00:42:28.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40220.log 2024-09-18T00:42:28.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114753.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67068.log.gz 2024-09-18T00:42:28.953 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37118.log 2024-09-18T00:42:28.953 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95201.log 2024-09-18T00:42:28.953 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41692.log 2024-09-18T00:42:28.953 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.95201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39150.log 2024-09-18T00:42:28.953 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57353.log 2024-09-18T00:42:28.953 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39150.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123898.log 2024-09-18T00:42:28.953 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113764.log 2024-09-18T00:42:28.953 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142619.log 2024-09-18T00:42:28.953 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113764.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52090.log 2024-09-18T00:42:28.953 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142619.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73824.log 2024-09-18T00:42:28.953 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52090.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127730.log 2024-09-18T00:42:28.953 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73824.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88732.log 2024-09-18T00:42:28.953 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.127730.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121882.log 2024-09-18T00:42:28.954 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36659.log 2024-09-18T00:42:28.954 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95107.log 2024-09-18T00:42:28.954 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76717.log 2024-09-18T00:42:28.954 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95107.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44882.log 2024-09-18T00:42:28.954 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133919.log 2024-09-18T00:42:28.954 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44882.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116739.log 2024-09-18T00:42:28.954 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78525.log 2024-09-18T00:42:28.954 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.116739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126544.log 2024-09-18T00:42:28.954 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78525.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137028.log 2024-09-18T00:42:28.954 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126544.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127488.log 2024-09-18T00:42:28.954 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135767.log 2024-09-18T00:42:28.954 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113242.log 2024-09-18T00:42:28.954 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135767.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136754.log 2024-09-18T00:42:28.955 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113242.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130606.log 2024-09-18T00:42:28.955 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135934.log.gz 2024-09-18T00:42:28.955 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135895.log.gz 2024-09-18T00:42:28.955 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88730.log 2024-09-18T00:42:28.955 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135935.log.gz 2024-09-18T00:42:28.955 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.130606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63078.log 2024-09-18T00:42:28.955 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88730.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44510.log 2024-09-18T00:42:28.955 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63078.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95175.log 2024-09-18T00:42:28.955 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44510.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44510.log.gz 2024-09-18T00:42:28.955 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128043.log 2024-09-18T00:42:28.955 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112441.log 2024-09-18T00:42:28.955 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.128043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66924.log 2024-09-18T00:42:28.955 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112441.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121567.log 2024-09-18T00:42:28.956 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66924.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98738.log.gz 2024-09-18T00:42:28.956 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88181.log 2024-09-18T00:42:28.956 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91188.log 2024-09-18T00:42:28.956 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88181.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64215.log 2024-09-18T00:42:28.956 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62725.log 2024-09-18T00:42:28.956 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124914.log 2024-09-18T00:42:28.956 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62725.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77535.log 2024-09-18T00:42:28.956 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57270.log 2024-09-18T00:42:28.956 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40642.log 2024-09-18T00:42:28.956 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118643.log 2024-09-18T00:42:28.956 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40642.log: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.129217.log.gz 2024-09-18T00:42:28.956 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75572.log 2024-09-18T00:42:28.956 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.129237.log.gz 2024-09-18T00:42:28.957 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122803.log 2024-09-18T00:42:28.957 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145903.log 2024-09-18T00:42:28.957 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40759.log 2024-09-18T00:42:28.957 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79645.log 2024-09-18T00:42:28.957 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42957.log 2024-09-18T00:42:28.957 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124576.log 2024-09-18T00:42:28.957 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42957.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105424.log 2024-09-18T00:42:28.957 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125954.log 2024-09-18T00:42:28.957 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.105424.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105424.log.gz 2024-09-18T00:42:28.957 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53686.log.gz 2024-09-18T00:42:28.957 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124610.log 2024-09-18T00:42:28.957 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97782.log 2024-09-18T00:42:28.957 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74459.log 2024-09-18T00:42:28.958 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63794.log 2024-09-18T00:42:28.958 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74459.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112274.log 2024-09-18T00:42:28.958 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63794.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34315.log 2024-09-18T00:42:28.958 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146006.log 2024-09-18T00:42:28.958 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.34315.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34315.log.gz 2024-09-18T00:42:28.958 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119766.log 2024-09-18T00:42:28.958 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146006.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104642.log 2024-09-18T00:42:28.958 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119766.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57917.log 2024-09-18T00:42:28.958 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114165.log 2024-09-18T00:42:28.958 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57917.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130108.log 2024-09-18T00:42:28.958 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53642.log.gz 2024-09-18T00:42:28.958 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139149.log 2024-09-18T00:42:28.958 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.130108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63417.log 2024-09-18T00:42:28.958 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139149.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39403.log 2024-09-18T00:42:28.959 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63417.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43636.log 2024-09-18T00:42:28.959 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39403.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149683.log 2024-09-18T00:42:28.959 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67942.log 2024-09-18T00:42:28.959 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149683.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65288.log 2024-09-18T00:42:28.959 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36837.log 2024-09-18T00:42:28.959 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90601.log 2024-09-18T00:42:28.959 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45089.log 2024-09-18T00:42:28.959 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69401.log 2024-09-18T00:42:28.959 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.45089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91814.log 2024-09-18T00:42:28.959 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69401.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127330.log 2024-09-18T00:42:28.959 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122704.log 2024-09-18T00:42:28.959 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86036.log 2024-09-18T00:42:28.959 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46442.log 2024-09-18T00:42:28.960 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86036.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126502.log 2024-09-18T00:42:28.960 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46442.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46442.log.gz 2024-09-18T00:42:28.960 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141241.log 2024-09-18T00:42:28.960 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126502.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118068.log 2024-09-18T00:42:28.960 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94514.log 2024-09-18T00:42:28.960 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.118068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49772.log 2024-09-18T00:42:28.960 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.94514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52740.log 2024-09-18T00:42:28.960 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49068.log 2024-09-18T00:42:28.960 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71378.log 2024-09-18T00:42:28.960 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.49068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136904.log 2024-09-18T00:42:28.960 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71378.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74413.log 2024-09-18T00:42:28.960 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111812.log 2024-09-18T00:42:28.960 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114081.log 2024-09-18T00:42:28.961 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111812.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115942.log 2024-09-18T00:42:28.961 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52782.log 2024-09-18T00:42:28.961 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42535.log.gz 2024-09-18T00:42:28.961 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.115942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133729.log 2024-09-18T00:42:28.961 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67112.log 2024-09-18T00:42:28.961 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95536.log.gz 2024-09-18T00:42:28.961 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66690.log 2024-09-18T00:42:28.961 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.95617.log.gz 2024-09-18T00:42:28.961 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52198.log 2024-09-18T00:42:28.961 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66690.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87271.log 2024-09-18T00:42:28.961 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52198.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52550.log 2024-09-18T00:42:28.961 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120389.log 2024-09-18T00:42:28.961 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131716.log 2024-09-18T00:42:28.962 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106426.log 2024-09-18T00:42:28.962 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123140.log 2024-09-18T00:42:28.962 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.106426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67155.log 2024-09-18T00:42:28.962 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150536.log 2024-09-18T00:42:28.962 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145667.log 2024-09-18T00:42:28.962 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150536.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139874.log 2024-09-18T00:42:28.962 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143710.log 2024-09-18T00:42:28.962 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113741.log.gz 2024-09-18T00:42:28.962 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92941.log 2024-09-18T00:42:28.962 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56033.log 2024-09-18T00:42:28.962 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92941.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94245.log 2024-09-18T00:42:28.962 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56033.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.56033.log.gz 2024-09-18T00:42:28.962 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67887.log 2024-09-18T00:42:28.963 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.94245.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99996.log 2024-09-18T00:42:28.963 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.67887.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.67887.log.gz 2024-09-18T00:42:28.963 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66228.log 2024-09-18T00:42:28.963 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99996.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113328.log.gz 2024-09-18T00:42:28.963 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141750.log 2024-09-18T00:42:28.963 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66228.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113451.log.gz 2024-09-18T00:42:28.963 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77431.log 2024-09-18T00:42:28.963 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75941.log 2024-09-18T00:42:28.963 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123687.log 2024-09-18T00:42:28.963 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122593.log 2024-09-18T00:42:28.963 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52888.log 2024-09-18T00:42:28.963 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122593.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79594.log 2024-09-18T00:42:28.963 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52888.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139409.log 2024-09-18T00:42:28.964 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79594.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92073.log 2024-09-18T00:42:28.964 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88394.log 2024-09-18T00:42:28.964 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92073.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33937.log 2024-09-18T00:42:28.964 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88394.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107993.log 2024-09-18T00:42:28.964 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33937.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146843.log.gz 2024-09-18T00:42:28.964 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33937.log.gz 2024-09-18T00:42:28.964 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138054.log 2024-09-18T00:42:28.964 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36775.log 2024-09-18T00:42:28.964 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138054.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107394.log 2024-09-18T00:42:28.964 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36775.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60602.log 2024-09-18T00:42:28.964 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107394.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64212.log 2024-09-18T00:42:28.964 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149782.log 2024-09-18T00:42:28.964 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117752.log 2024-09-18T00:42:28.965 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95420.log 2024-09-18T00:42:28.965 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.117752.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.90602.log.gz 2024-09-18T00:42:28.965 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120024.log 2024-09-18T00:42:28.965 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95420.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82789.log 2024-09-18T00:42:28.965 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125559.log 2024-09-18T00:42:28.965 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.82789.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78626.log.gz 2024-09-18T00:42:28.965 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90284.log 2024-09-18T00:42:28.965 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125559.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50997.log 2024-09-18T00:42:28.965 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90284.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122471.log 2024-09-18T00:42:28.965 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139740.log 2024-09-18T00:42:28.965 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148228.log 2024-09-18T00:42:28.965 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84899.log 2024-09-18T00:42:28.965 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90974.log 2024-09-18T00:42:28.966 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84899.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153701.log 2024-09-18T00:42:28.966 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149473.log 2024-09-18T00:42:28.966 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.153701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150414.log 2024-09-18T00:42:28.966 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149473.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90091.log.gz 2024-09-18T00:42:28.966 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111643.log 2024-09-18T00:42:28.966 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150414.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87020.log 2024-09-18T00:42:28.966 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104180.log 2024-09-18T00:42:28.966 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131618.log 2024-09-18T00:42:28.966 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104180.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109229.log 2024-09-18T00:42:28.966 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131618.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79648.log 2024-09-18T00:42:28.966 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1%/var/log/ceph/cluster1-client.mirror.109229.log: -- replaced with /var/log/ceph/cluster1-client.mirror.102266.log.gz 2024-09-18T00:42:28.966 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43702.log 2024-09-18T00:42:28.966 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79648.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.79648.log.gz 2024-09-18T00:42:28.966 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66733.log 2024-09-18T00:42:28.967 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36881.log 2024-09-18T00:42:28.967 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66733.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83581.log 2024-09-18T00:42:28.967 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.36881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40180.log 2024-09-18T00:42:28.967 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62408.log 2024-09-18T00:42:28.967 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40180.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146299.log 90.0% 2024-09-18T00:42:28.967 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.102350.log.gz 2024-09-18T00:42:28.967 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62408.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107263.log 2024-09-18T00:42:28.967 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146299.log: 96.3%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.153701.log.gz 2024-09-18T00:42:28.967 INFO:teuthology.orchestra.run.smithi192.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.152229.log 2024-09-18T00:42:28.967 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.107263.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43962.log 2024-09-18T00:42:28.967 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152229.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42532.log 2024-09-18T00:42:28.967 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43962.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54526.log 2024-09-18T00:42:28.967 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42532.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46822.log 2024-09-18T00:42:28.968 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132176.log 2024-09-18T00:42:28.968 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.46822.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117410.log 2024-09-18T00:42:28.968 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.132176.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129264.log 2024-09-18T00:42:28.968 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89379.log.gz 2024-09-18T00:42:28.968 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.117410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111434.log 2024-09-18T00:42:28.968 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129264.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124442.log 2024-09-18T00:42:28.968 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111434.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109961.log 2024-09-18T00:42:28.968 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91870.log 2024-09-18T00:42:28.968 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100252.log 2024-09-18T00:42:28.968 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98779.log 2024-09-18T00:42:28.968 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92193.log 2024-09-18T00:42:28.968 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108493.log.gz 2024-09-18T00:42:28.968 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98779.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63144.log 2024-09-18T00:42:28.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75422.log 2024-09-18T00:42:28.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133962.log 2024-09-18T00:42:28.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75422.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62724.log 2024-09-18T00:42:28.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133962.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47050.log 2024-09-18T00:42:28.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66925.log 2024-09-18T00:42:28.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.47050.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114371.log 2024-09-18T00:42:28.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66925.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76273.log 2024-09-18T00:42:28.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98277.log 2024-09-18T00:42:28.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38940.log 2024-09-18T00:42:28.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98277.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68473.log 2024-09-18T00:42:28.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107112.log 2024-09-18T00:42:28.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68473.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67750.log 2024-09-18T00:42:28.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.107112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110172.log 2024-09-18T00:42:28.970 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60168.log 2024-09-18T00:42:28.970 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109390.log.gz 2024-09-18T00:42:28.970 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35215.log 2024-09-18T00:42:28.970 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60168.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105398.log 2024-09-18T00:42:28.970 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.35215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65137.log 2024-09-18T00:42:28.970 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.105398.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105398.log.gz 2024-09-18T00:42:28.970 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140214.log 2024-09-18T00:42:28.970 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123727.log 2024-09-18T00:42:28.970 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.140214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108297.log 2024-09-18T00:42:28.970 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56387.log 2024-09-18T00:42:28.970 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108297.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76039.log.gz 2024-09-18T00:42:28.970 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70147.log 2024-09-18T00:42:28.970 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76005.log.gz 2024-09-18T00:42:28.970 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78328.log 2024-09-18T00:42:28.971 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70147.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147326.log 2024-09-18T00:42:28.971 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133247.log 2024-09-18T00:42:28.971 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147326.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129404.log 2024-09-18T00:42:28.971 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133247.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116674.log 2024-09-18T00:42:28.971 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.147326.log.gz/var/log/ceph/cluster2-client.mirror.129404.log: 2024-09-18T00:42:28.971 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.127488.log.gz 2024-09-18T00:42:28.971 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35351.log 2024-09-18T00:42:28.971 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.116674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56827.log 2024-09-18T00:42:29.107 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.35351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136879.log 2024-09-18T00:42:29.107 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56827.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142537.log 2024-09-18T00:42:29.107 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136879.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80620.log 2024-09-18T00:42:29.107 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51189.log 2024-09-18T00:42:29.107 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.80620.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54148.log 2024-09-18T00:42:29.107 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51189.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127840.log 2024-09-18T00:42:29.107 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54148.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52028.log 2024-09-18T00:42:29.107 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.127840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43073.log 2024-09-18T00:42:29.108 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75718.log 2024-09-18T00:42:29.108 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104417.log 2024-09-18T00:42:29.108 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1%/var/log/ceph/cluster1-client.mirror.75718.log: -- replaced with /var/log/ceph/cluster1-client.mirror.40549.log.gz 2024-09-18T00:42:29.108 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131639.log 2024-09-18T00:42:29.108 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.104417.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55577.log 2024-09-18T00:42:29.108 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131639.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98167.log 2024-09-18T00:42:29.108 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55577.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51943.log 2024-09-18T00:42:29.108 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98167.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41863.log 2024-09-18T00:42:29.108 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51943.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128932.log 2024-09-18T00:42:29.108 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41863.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73825.log 2024-09-18T00:42:29.108 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128932.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128932.log.gz 2024-09-18T00:42:29.108 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146906.log 2024-09-18T00:42:29.108 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73825.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62323.log.gz 2024-09-18T00:42:29.109 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147793.log 2024-09-18T00:42:29.109 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111391.log 2024-09-18T00:42:29.109 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112486.log 2024-09-18T00:42:29.109 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88582.log 2024-09-18T00:42:29.109 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112486.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90389.log 2024-09-18T00:42:29.109 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.147793.log.gz 2024-09-18T00:42:29.109 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135850.log 2024-09-18T00:42:29.109 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113117.log 2024-09-18T00:42:29.109 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39420.log 2024-09-18T00:42:29.109 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113117.log: gzip -5 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.129404.log.gz 2024-09-18T00:42:29.109 INFO:teuthology.orchestra.run.smithi192.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.88814.log 2024-09-18T00:42:29.109 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39420.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130391.log 2024-09-18T00:42:29.109 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41140.log 2024-09-18T00:42:29.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125282.log 2024-09-18T00:42:29.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39606.log 2024-09-18T00:42:29.110 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53517.log.gz 2024-09-18T00:42:29.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125282.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45450.log 2024-09-18T00:42:29.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148586.log 2024-09-18T00:42:29.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45450.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.107112.log.gz 2024-09-18T00:42:29.110 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117658.log 2024-09-18T00:42:29.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148586.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99997.log 2024-09-18T00:42:29.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.117658.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112485.log 2024-09-18T00:42:29.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128666.log 2024-09-18T00:42:29.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45840.log 2024-09-18T00:42:29.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.128666.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87022.log 2024-09-18T00:42:29.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123834.log 2024-09-18T00:42:29.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110803.log 2024-09-18T00:42:29.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82250.log 2024-09-18T00:42:29.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91687.log 2024-09-18T00:42:29.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.82250.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33334.log 2024-09-18T00:42:29.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47548.log 2024-09-18T00:42:29.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73928.log 2024-09-18T00:42:29.111 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33334.log.gz 2024-09-18T00:42:29.111 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124509.log.gz 2024-09-18T00:42:29.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.47548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148694.log 2024-09-18T00:42:29.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73928.log: /var/log/ceph/cluster1-client.mirror.148694.log: gzip -5 --verbose 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.118208.log.gz 2024-09-18T00:42:29.111 INFO:teuthology.orchestra.run.smithi192.stderr: -- /var/log/ceph/cluster1-client.mirror.63336.log 2024-09-18T00:42:29.111 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89320.log 2024-09-18T00:42:29.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63336.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82437.log 2024-09-18T00:42:29.112 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122849.log 2024-09-18T00:42:29.112 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.82437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65599.log 2024-09-18T00:42:29.112 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.127130.log.gz 2024-09-18T00:42:29.112 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122849.log: gzip -5 --verbose -- 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.82250.log.gz 2024-09-18T00:42:29.112 INFO:teuthology.orchestra.run.smithi192.stderr: /var/log/ceph/cluster1-client.admin.139041.log 2024-09-18T00:42:29.112 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65599.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96429.log 2024-09-18T00:42:29.112 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139041.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127127.log 2024-09-18T00:42:29.112 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148103.log 2024-09-18T00:42:29.112 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127127.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59865.log 2024-09-18T00:42:29.112 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147685.log 2024-09-18T00:42:29.112 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59865.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149574.log 2024-09-18T00:42:29.112 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111602.log.gz 2024-09-18T00:42:29.112 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.115369.log.gz 2024-09-18T00:42:29.113 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130805.log 2024-09-18T00:42:29.113 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58868.log 2024-09-18T00:42:29.113 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64968.log 2024-09-18T00:42:29.113 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.58868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112275.log 2024-09-18T00:42:29.113 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133459.log 2024-09-18T00:42:29.113 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41602.log 2024-09-18T00:42:29.113 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133459.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77155.log 2024-09-18T00:42:29.113 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148945.log 2024-09-18T00:42:29.113 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50912.log 2024-09-18T00:42:29.113 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144936.log 2024-09-18T00:42:29.113 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55512.log 2024-09-18T00:42:29.113 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144936.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92788.log 2024-09-18T00:42:29.114 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55512.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152901.log 2024-09-18T00:42:29.114 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.92788.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149742.log 2024-09-18T00:42:29.114 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.152901.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105567.log 2024-09-18T00:42:29.114 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95384.log 2024-09-18T00:42:29.114 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.105567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124273.log 2024-09-18T00:42:29.114 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53179.log 2024-09-18T00:42:29.114 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100546.log 2024-09-18T00:42:29.114 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132475.log 2024-09-18T00:42:29.114 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146593.log 2024-09-18T00:42:29.114 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.132475.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139632.log 2024-09-18T00:42:29.114 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146593.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119370.log 2024-09-18T00:42:29.114 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139632.log: gzip -5 --verbose -- 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146593.log.gz 2024-09-18T00:42:29.115 INFO:teuthology.orchestra.run.smithi192.stderr: /var/log/ceph/cluster1-client.admin.95488.log 2024-09-18T00:42:29.115 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55365.log.gz 2024-09-18T00:42:29.115 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119370.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.152901.log.gz 2024-09-18T00:42:29.115 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131819.log 2024-09-18T00:42:29.115 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.95488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42652.log 2024-09-18T00:42:29.115 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131819.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71272.log.gz 2024-09-18T00:42:29.115 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41902.log 2024-09-18T00:42:29.115 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113745.log 2024-09-18T00:42:29.115 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135744.log 2024-09-18T00:42:29.115 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113745.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139936.log 2024-09-18T00:42:29.115 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135744.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81180.log 2024-09-18T00:42:29.115 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139936.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70785.log 2024-09-18T00:42:29.115 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.81180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-18T00:42:29.116 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60602.log.gz 2024-09-18T00:42:29.116 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117565.log 2024-09-18T00:42:29.116 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131216.log 2024-09-18T00:42:29.116 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.117565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54778.log 2024-09-18T00:42:29.116 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.131216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106520.log 2024-09-18T00:42:29.116 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54778.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125789.log 2024-09-18T00:42:29.116 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86037.log 2024-09-18T00:42:29.116 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125789.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112402.log 2024-09-18T00:42:29.116 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86037.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106894.log 2024-09-18T00:42:29.116 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112402.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122848.log.gz 2024-09-18T00:42:29.116 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145693.log 2024-09-18T00:42:29.116 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106894.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.150830.log 2024-09-18T00:42:29.117 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106198.log 2024-09-18T00:42:29.117 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.150830.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126165.log 2024-09-18T00:42:29.117 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.106198.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101700.log 2024-09-18T00:42:29.117 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126165.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111392.log.gz 2024-09-18T00:42:29.117 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69101.log 2024-09-18T00:42:29.117 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119191.log 2024-09-18T00:42:29.117 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50911.log 2024-09-18T00:42:29.117 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99366.log 2024-09-18T00:42:29.117 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42023.log 2024-09-18T00:42:29.117 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149426.log 2024-09-18T00:42:29.117 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42023.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63203.log 2024-09-18T00:42:29.117 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124274.log.gz 2024-09-18T00:42:29.117 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110132.log 2024-09-18T00:42:29.117 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63203.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.91870.log.gz 2024-09-18T00:42:29.118 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93997.log 2024-09-18T00:42:29.118 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69643.log 2024-09-18T00:42:29.118 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.93997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140682.log 2024-09-18T00:42:29.118 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80820.log 2024-09-18T00:42:29.118 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140682.log: gzip -5 --verbose -- 0.0% /var/log/ceph/cluster1-client.mirror.39760.log 2024-09-18T00:42:29.118 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.140682.log.gz 2024-09-18T00:42:29.118 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59671.log 2024-09-18T00:42:29.118 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133543.log 2024-09-18T00:42:29.118 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74372.log 2024-09-18T00:42:29.118 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133543.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133311.log 2024-09-18T00:42:29.118 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148903.log 2024-09-18T00:42:29.118 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133311.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105200.log 2024-09-18T00:42:29.118 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113533.log 2024-09-18T00:42:29.119 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.105200.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81042.log 2024-09-18T00:42:29.119 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78942.log 2024-09-18T00:42:29.119 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81042.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112483.log 2024-09-18T00:42:29.119 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88391.log 2024-09-18T00:42:29.119 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114502.log.gz 2024-09-18T00:42:29.119 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126123.log 2024-09-18T00:42:29.119 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109546.log 2024-09-18T00:42:29.119 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126123.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75002.log 2024-09-18T00:42:29.119 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132743.log 2024-09-18T00:42:29.119 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49583.log 2024-09-18T00:42:29.119 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122408.log.gz 2024-09-18T00:42:29.119 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.132743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79994.log 2024-09-18T00:42:29.119 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61992.log 2024-09-18T00:42:29.120 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128449.log 2024-09-18T00:42:29.120 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.61992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121335.log 2024-09-18T00:42:29.120 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.128449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123980.log 2024-09-18T00:42:29.120 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60806.log 2024-09-18T00:42:29.120 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123980.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119937.log 2024-09-18T00:42:29.120 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60806.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110971.log 2024-09-18T00:42:29.120 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119937.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55700.log 2024-09-18T00:42:29.120 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.131415.log.gz 2024-09-18T00:42:29.120 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77768.log 2024-09-18T00:42:29.120 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55700.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104216.log 2024-09-18T00:42:29.120 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36423.log 2024-09-18T00:42:29.120 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89150.log 2024-09-18T00:42:29.121 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.36423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36965.log 2024-09-18T00:42:29.121 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89150.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34231.log 2024-09-18T00:42:29.121 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36965.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.142467.log.gz 2024-09-18T00:42:29.121 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48635.log 2024-09-18T00:42:29.121 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.34231.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149995.log 2024-09-18T00:42:29.121 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34231.log.gz 2024-09-18T00:42:29.121 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79089.log 2024-09-18T00:42:29.121 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100207.log 2024-09-18T00:42:29.121 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64910.log 2024-09-18T00:42:29.121 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100207.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122384.log 2024-09-18T00:42:29.121 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91408.log 2024-09-18T00:42:29.121 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122384.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45349.log 2024-09-18T00:42:29.121 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91408.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91408.log.gz 2024-09-18T00:42:29.122 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42493.log 2024-09-18T00:42:29.122 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45349.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100650.log 2024-09-18T00:42:29.122 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71972.log 2024-09-18T00:42:29.122 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70738.log.gz 2024-09-18T00:42:29.122 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100650.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98232.log 2024-09-18T00:42:29.122 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151332.log 2024-09-18T00:42:29.122 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64255.log 2024-09-18T00:42:29.122 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60628.log 2024-09-18T00:42:29.122 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93971.log 2024-09-18T00:42:29.122 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141066.log 2024-09-18T00:42:29.122 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.93971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100253.log 2024-09-18T00:42:29.122 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123392.log 2024-09-18T00:42:29.123 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109754.log 2024-09-18T00:42:29.123 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123392.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48546.log.gz 2024-09-18T00:42:29.123 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62617.log 2024-09-18T00:42:29.123 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125753.log 2024-09-18T00:42:29.123 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62617.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57385.log 2024-09-18T00:42:29.123 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102643.log 2024-09-18T00:42:29.123 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51290.log 2024-09-18T00:42:29.123 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126209.log 2024-09-18T00:42:29.123 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88732.log.gz/var/log/ceph/cluster1-client.mirror.51290.log: 2024-09-18T00:42:29.123 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63917.log 2024-09-18T00:42:29.123 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115306.log 2024-09-18T00:42:29.123 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63917.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47320.log 2024-09-18T00:42:29.123 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79803.log 2024-09-18T00:42:29.124 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.47320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113872.log 2024-09-18T00:42:29.124 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79803.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68869.log 2024-09-18T00:42:29.124 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113872.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35167.log 2024-09-18T00:42:29.124 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.68869.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134929.log 2024-09-18T00:42:29.124 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52382.log.gz 2024-09-18T00:42:29.124 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.35167.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35167.log.gz 2024-09-18T00:42:29.124 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50163.log 2024-09-18T00:42:29.124 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134929.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124200.log 2024-09-18T00:42:29.124 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72470.log 2024-09-18T00:42:29.124 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124200.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58686.log 2024-09-18T00:42:29.124 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72470.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145037.log 2024-09-18T00:42:29.124 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.58686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138163.log 2024-09-18T00:42:29.124 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145037.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108852.log 2024-09-18T00:42:29.125 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105769.log 2024-09-18T00:42:29.125 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140183.log 2024-09-18T00:42:29.125 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.105769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138014.log 2024-09-18T00:42:29.125 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.140183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100377.log 2024-09-18T00:42:29.125 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42116.log 2024-09-18T00:42:29.125 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100377.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39549.log 2024-09-18T00:42:29.125 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42116.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44151.log 2024-09-18T00:42:29.125 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39549.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46729.log 2024-09-18T00:42:29.125 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44151.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123560.log 2024-09-18T00:42:29.125 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.46729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105201.log 2024-09-18T00:42:29.125 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123560.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136817.log 2024-09-18T00:42:29.125 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.105201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39629.log 2024-09-18T00:42:29.125 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150450.log 2024-09-18T00:42:29.126 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39629.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47770.log 2024-09-18T00:42:29.126 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150450.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103044.log 2024-09-18T00:42:29.126 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118607.log 2024-09-18T00:42:29.126 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.103044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132416.log 2024-09-18T00:42:29.126 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118607.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89800.log 2024-09-18T00:42:29.126 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.132416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149953.log 2024-09-18T00:42:29.126 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66063.log 2024-09-18T00:42:29.126 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.154611.log 2024-09-18T00:42:29.126 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66063.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36684.log 2024-09-18T00:42:29.126 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.154611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59575.log 2024-09-18T00:42:29.126 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.154611.log.gz 2024-09-18T00:42:29.126 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36684.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143194.log 2024-09-18T00:42:29.126 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91543.log 2024-09-18T00:42:29.127 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37702.log 2024-09-18T00:42:29.127 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91543.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52467.log 2024-09-18T00:42:29.127 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91543.log.gz 2024-09-18T00:42:29.127 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37702.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.153879.log 2024-09-18T00:42:29.127 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52467.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73146.log 2024-09-18T00:42:29.127 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.153879.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.153879.log.gz 2024-09-18T00:42:29.127 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133247.log.gz 2024-09-18T00:42:29.127 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65472.log 2024-09-18T00:42:29.127 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57479.log 2024-09-18T00:42:29.127 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58210.log 2024-09-18T00:42:29.127 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.57479.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129310.log 2024-09-18T00:42:29.127 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.58210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52195.log 2024-09-18T00:42:29.127 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54022.log 2024-09-18T00:42:29.127 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81597.log 2024-09-18T00:42:29.128 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94855.log 2024-09-18T00:42:29.128 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81597.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81597.log.gz 2024-09-18T00:42:29.128 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152036.log 2024-09-18T00:42:29.128 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.94855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78710.log 2024-09-18T00:42:29.128 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152036.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109599.log 2024-09-18T00:42:29.128 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60190.log 2024-09-18T00:42:29.128 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109599.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67530.log 2024-09-18T00:42:29.128 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38726.log 2024-09-18T00:42:29.128 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67530.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65829.log 2024-09-18T00:42:29.128 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38726.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56004.log 2024-09-18T00:42:29.128 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65829.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91423.log 2024-09-18T00:42:29.129 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56004.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.56004.log.gz 2024-09-18T00:42:29.129 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125216.log 2024-09-18T00:42:29.129 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91423.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74413.log.gz 2024-09-18T00:42:29.129 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84083.log 2024-09-18T00:42:29.129 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120491.log 2024-09-18T00:42:29.129 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.84083.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78103.log 2024-09-18T00:42:29.129 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120491.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86890.log 2024-09-18T00:42:29.129 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102852.log 2024-09-18T00:42:29.129 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77810.log 2024-09-18T00:42:29.129 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102852.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92271.log 2024-09-18T00:42:29.129 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77810.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82411.log 2024-09-18T00:42:29.129 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49583.log.gz 2024-09-18T00:42:29.129 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130035.log 2024-09-18T00:42:29.130 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.82411.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105108.log 2024-09-18T00:42:29.130 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.130035.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148228.log.gz 2024-09-18T00:42:29.130 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63375.log 2024-09-18T00:42:29.130 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.105108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87320.log 2024-09-18T00:42:29.130 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51188.log 2024-09-18T00:42:29.130 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89001.log 2024-09-18T00:42:29.130 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122678.log 2024-09-18T00:42:29.130 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98988.log 2024-09-18T00:42:29.130 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122678.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124719.log 2024-09-18T00:42:29.130 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75170.log.gz 2024-09-18T00:42:29.130 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98988.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33427.log 2024-09-18T00:42:29.130 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67110.log 2024-09-18T00:42:29.131 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33427.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70644.log 2024-09-18T00:42:29.131 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33427.log.gz 2024-09-18T00:42:29.131 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53600.log 2024-09-18T00:42:29.131 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110131.log 2024-09-18T00:42:29.131 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53600.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124108.log 2024-09-18T00:42:29.131 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41693.log 2024-09-18T00:42:29.131 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108142.log 2024-09-18T00:42:29.131 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121379.log 2024-09-18T00:42:29.131 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108142.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104587.log 2024-09-18T00:42:29.131 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71946.log 2024-09-18T00:42:29.131 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67155.log.gz 2024-09-18T00:42:29.131 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.104587.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105489.log 2024-09-18T00:42:29.131 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71946.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51626.log 2024-09-18T00:42:29.132 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.105489.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105489.log.gz 2024-09-18T00:42:29.132 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45751.log 2024-09-18T00:42:29.132 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146633.log 2024-09-18T00:42:29.132 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126081.log 2024-09-18T00:42:29.132 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146633.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84328.log 2024-09-18T00:42:29.132 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116362.log 2024-09-18T00:42:29.132 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.146633.log.gz 2024-09-18T00:42:29.132 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70188.log 2024-09-18T00:42:29.132 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88731.log 2024-09-18T00:42:29.132 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90013.log 2024-09-18T00:42:29.132 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88731.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48564.log 2024-09-18T00:42:29.132 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144933.log 2024-09-18T00:42:29.132 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58230.log 2024-09-18T00:42:29.132 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114711.log 2024-09-18T00:42:29.133 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.58230.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71080.log 2024-09-18T00:42:29.133 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114711.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143670.log 2024-09-18T00:42:29.133 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71080.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55997.log 2024-09-18T00:42:29.133 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135766.log 2024-09-18T00:42:29.133 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55997.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72430.log 2024-09-18T00:42:29.133 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120841.log 2024-09-18T00:42:29.133 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147834.log 2024-09-18T00:42:29.133 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74416.log 2024-09-18T00:42:29.133 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54358.log 2024-09-18T00:42:29.133 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111644.log 2024-09-18T00:42:29.133 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54358.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147434.log 2024-09-18T00:42:29.133 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.147834.log.gz 2024-09-18T00:42:29.134 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88182.log 2024-09-18T00:42:29.134 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147434.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98862.log 2024-09-18T00:42:29.134 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67734.log 2024-09-18T00:42:29.134 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98862.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.147434.log.gz 2024-09-18T00:42:29.134 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67846.log 2024-09-18T00:42:29.134 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.67734.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.67734.log.gz 2024-09-18T00:42:29.134 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39635.log 2024-09-18T00:42:29.134 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35977.log 2024-09-18T00:42:29.134 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35835.log 2024-09-18T00:42:29.134 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.35977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135430.log 2024-09-18T00:42:29.134 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.35835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106084.log 2024-09-18T00:42:29.134 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152689.log 2024-09-18T00:42:29.134 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.106084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51669.log 2024-09-18T00:42:29.135 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77744.log 2024-09-18T00:42:29.135 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.152689.log.gz 2024-09-18T00:42:29.135 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76296.log 2024-09-18T00:42:29.135 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77744.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122470.log 2024-09-18T00:42:29.135 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76296.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111181.log 2024-09-18T00:42:29.135 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122470.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95038.log 2024-09-18T00:42:29.135 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111181.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122299.log 2024-09-18T00:42:29.135 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95038.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89088.log 2024-09-18T00:42:29.135 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78145.log 2024-09-18T00:42:29.135 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45881.log 2024-09-18T00:42:29.135 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78145.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115156.log 2024-09-18T00:42:29.135 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.45881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90511.log 2024-09-18T00:42:29.135 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115156.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.115156.log.gz 2024-09-18T00:42:29.136 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89650.log 2024-09-18T00:42:29.136 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108163.log 2024-09-18T00:42:29.136 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89650.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76569.log 2024-09-18T00:42:29.136 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63416.log 2024-09-18T00:42:29.136 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76569.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133394.log 2024-09-18T00:42:29.136 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63416.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58552.log 2024-09-18T00:42:29.136 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133394.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100819.log 2024-09-18T00:42:29.136 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.58552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51942.log 2024-09-18T00:42:29.136 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103216.log 2024-09-18T00:42:29.136 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124358.log 2024-09-18T00:42:29.136 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103216.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.103216.log.gz 2024-09-18T00:42:29.136 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120233.log 2024-09-18T00:42:29.136 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124358.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41233.log 2024-09-18T00:42:29.137 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120233.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97924.log 2024-09-18T00:42:29.137 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41233.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115766.log 2024-09-18T00:42:29.137 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97924.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117227.log 2024-09-18T00:42:29.137 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.115766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153400.log 2024-09-18T00:42:29.137 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.117227.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.117227.log.gz 2024-09-18T00:42:29.137 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37253.log 2024-09-18T00:42:29.137 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.153400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64739.log 2024-09-18T00:42:29.137 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37253.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.153400.log.gz 2024-09-18T00:42:29.137 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41065.log 2024-09-18T00:42:29.137 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71828.log 2024-09-18T00:42:29.137 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112276.log 2024-09-18T00:42:29.137 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71828.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102142.log 2024-09-18T00:42:29.137 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112276.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56557.log 2024-09-18T00:42:29.138 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101256.log/var/log/ceph/cluster1-client.mirror.102142.log: 2024-09-18T00:42:29.138 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110385.log 2024-09-18T00:42:29.138 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103201.log 2024-09-18T00:42:29.138 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62868.log 2024-09-18T00:42:29.138 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99661.log 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.103201.log.gz 2024-09-18T00:42:29.138 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:29.138 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142119.log 2024-09-18T00:42:29.138 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70831.log 2024-09-18T00:42:29.138 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.142119.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42325.log 2024-09-18T00:42:29.138 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48482.log 2024-09-18T00:42:29.138 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77240.log 2024-09-18T00:42:29.138 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52784.log 2024-09-18T00:42:29.138 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77240.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39970.log 2024-09-18T00:42:29.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52784.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57653.log 2024-09-18T00:42:29.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91617.log 2024-09-18T00:42:29.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146784.log 2024-09-18T00:42:29.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55405.log 2024-09-18T00:42:29.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146784.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97647.log 2024-09-18T00:42:29.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55405.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115601.log 2024-09-18T00:42:29.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115453.log 2024-09-18T00:42:29.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115601.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146784.log.gz 2024-09-18T00:42:29.139 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64590.log 2024-09-18T00:42:29.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.115453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60766.log 2024-09-18T00:42:29.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64590.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41486.log 2024-09-18T00:42:29.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50702.log 2024-09-18T00:42:29.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41486.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64674.log 2024-09-18T00:42:29.140 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50702.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.154226.log 2024-09-18T00:42:29.140 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76194.log 2024-09-18T00:42:29.140 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.154226.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.154226.log.gz 2024-09-18T00:42:29.140 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78880.log 2024-09-18T00:42:29.140 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76194.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117001.log 2024-09-18T00:42:29.140 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82270.log 2024-09-18T00:42:29.140 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.117001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114791.log 2024-09-18T00:42:29.140 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.82270.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47279.log 2024-09-18T00:42:29.140 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124823.log 2024-09-18T00:42:29.140 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.47279.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107222.log 2024-09-18T00:42:29.140 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75131.log.gz 2024-09-18T00:42:29.140 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124823.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118114.log 2024-09-18T00:42:29.140 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41485.log 2024-09-18T00:42:29.141 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.118114.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103588.log 2024-09-18T00:42:29.141 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35522.log 2024-09-18T00:42:29.141 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103588.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51944.log 2024-09-18T00:42:29.141 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.35522.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136314.log.gz 2024-09-18T00:42:29.141 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75003.log 2024-09-18T00:42:29.141 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143041.log 2024-09-18T00:42:29.141 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130959.log 2024-09-18T00:42:29.141 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143041.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54062.log 2024-09-18T00:42:29.141 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54441.log 2024-09-18T00:42:29.141 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61173.log 2024-09-18T00:42:29.141 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54441.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141961.log 2024-09-18T00:42:29.141 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.61173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92194.log 2024-09-18T00:42:29.141 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79645.log.gz 2024-09-18T00:42:29.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.141961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39759.log 2024-09-18T00:42:29.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147371.log 2024-09-18T00:42:29.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141941.log 2024-09-18T00:42:29.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104432.log 2024-09-18T00:42:29.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.141941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98652.log 2024-09-18T00:42:29.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.104432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34945.log 2024-09-18T00:42:29.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80177.log 2024-09-18T00:42:29.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.34945.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34945.log.gz 2024-09-18T00:42:29.142 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98630.log 2024-09-18T00:42:29.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.80177.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78736.log.gz 2024-09-18T00:42:29.142 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87459.log 2024-09-18T00:42:29.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123098.log 2024-09-18T00:42:29.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87459.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67448.log 2024-09-18T00:42:29.143 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60341.log 2024-09-18T00:42:29.143 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132837.log 2024-09-18T00:42:29.143 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50911.log.gz 2024-09-18T00:42:29.143 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78881.log 2024-09-18T00:42:29.143 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.132837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94628.log 2024-09-18T00:42:29.143 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47835.log 2024-09-18T00:42:29.143 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.94628.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100123.log 2024-09-18T00:42:29.143 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83091.log 2024-09-18T00:42:29.143 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100123.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107921.log 2024-09-18T00:42:29.143 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89150.log.gz 2024-09-18T00:42:29.143 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60496.log 2024-09-18T00:42:29.143 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107921.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70717.log 2024-09-18T00:42:29.143 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59210.log 2024-09-18T00:42:29.143 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89950.log 2024-09-18T00:42:29.144 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.59210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136020.log 2024-09-18T00:42:29.144 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89950.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128666.log.gz 2024-09-18T00:42:29.144 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149322.log 2024-09-18T00:42:29.144 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136020.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128974.log 2024-09-18T00:42:29.144 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113452.log 2024-09-18T00:42:29.144 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128974.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130690.log 2024-09-18T00:42:29.144 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.128974.log.gz 2024-09-18T00:42:29.144 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113452.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108094.log 2024-09-18T00:42:29.144 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130690.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111436.log 2024-09-18T00:42:29.144 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48979.log 2024-09-18T00:42:29.144 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111436.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114332.log 2024-09-18T00:42:29.144 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134233.log 2024-09-18T00:42:29.144 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114332.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117638.log 2024-09-18T00:42:29.145 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40432.log.gz 2024-09-18T00:42:29.145 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134233.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40429.log.gz 2024-09-18T00:42:29.145 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82209.log 2024-09-18T00:42:29.145 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.117638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82849.log 2024-09-18T00:42:29.145 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.82209.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94815.log 2024-09-18T00:42:29.145 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.82849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153380.log 2024-09-18T00:42:29.145 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.94815.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39211.log.gz 2024-09-18T00:42:29.145 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96097.log 2024-09-18T00:42:29.145 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.153380.log: gzip 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.153380.log.gz 2024-09-18T00:42:29.145 INFO:teuthology.orchestra.run.smithi192.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130320.log 2024-09-18T00:42:29.145 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96097.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43703.log 2024-09-18T00:42:29.145 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56445.log 2024-09-18T00:42:29.145 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43703.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94131.log 2024-09-18T00:42:29.146 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56445.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69404.log 2024-09-18T00:42:29.146 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.94131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113325.log 2024-09-18T00:42:29.146 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69404.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133729.log.gz 2024-09-18T00:42:29.146 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86584.log 2024-09-18T00:42:29.146 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83888.log 2024-09-18T00:42:29.146 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111011.log 2024-09-18T00:42:29.146 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83888.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112611.log 2024-09-18T00:42:29.146 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111011.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112067.log 2024-09-18T00:42:29.146 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112611.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34063.log 2024-09-18T00:42:29.146 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133854.log.gz 2024-09-18T00:42:29.146 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112067.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57914.log 2024-09-18T00:42:29.146 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.34063.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34063.log.gz 2024-09-18T00:42:29.146 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112481.log 2024-09-18T00:42:29.147 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57914.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54819.log 2024-09-18T00:42:29.147 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114581.log 2024-09-18T00:42:29.147 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112401.log 2024-09-18T00:42:29.147 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36621.log.gz 2024-09-18T00:42:29.147 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65475.log 2024-09-18T00:42:29.147 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112401.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110597.log 2024-09-18T00:42:29.147 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65475.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113114.log 2024-09-18T00:42:29.147 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110597.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63667.log 2024-09-18T00:42:29.147 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153245.log 2024-09-18T00:42:29.147 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117585.log 2024-09-18T00:42:29.147 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.153245.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104213.log 2024-09-18T00:42:29.147 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.117585.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49471.log 2024-09-18T00:42:29.147 INFO:teuthology.orchestra.run.smithi192.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119766.log.gz 2024-09-18T00:42:29.148 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81056.log 2024-09-18T00:42:29.148 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103723.log 2024-09-18T00:42:29.148 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76609.log 2024-09-18T00:42:29.148 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110387.log 2024-09-18T00:42:29.148 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135221.log 2024-09-18T00:42:29.205 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110387.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45844.log 2024-09-18T00:42:29.206 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99537.log 2024-09-18T00:42:29.206 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138373.log 2024-09-18T00:42:29.206 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99537.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114791.log.gz 2024-09-18T00:42:29.206 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151001.log 2024-09-18T00:42:29.206 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88314.log 2024-09-18T00:42:29.206 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151001.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.153245.log.gz 2024-09-18T00:42:29.206 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73145.log 2024-09-18T00:42:29.206 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66166.log 2024-09-18T00:42:29.206 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73145.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52029.log 2024-09-18T00:42:29.206 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66166.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98840.log 2024-09-18T00:42:29.206 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75824.log 2024-09-18T00:42:29.206 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98840.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152596.log 2024-09-18T00:42:29.207 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75824.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54992.log 2024-09-18T00:42:29.207 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152596.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.152596.log.gz 2024-09-18T00:42:29.207 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83896.log 2024-09-18T00:42:29.207 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103612.log 2024-09-18T00:42:29.207 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83896.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77684.log 2024-09-18T00:42:29.207 INFO:teuthology.orchestra.run.smithi192.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-client.admin.94313.log.gz 2024-09-18T00:42:29.207 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103612.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92676.log 2024-09-18T00:42:29.207 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128369.log 2024-09-18T00:42:29.207 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37999.log 2024-09-18T00:42:29.207 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128369.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.127127.log.gz 2024-09-18T00:42:29.207 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138794.log 2024-09-18T00:42:29.208 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41485.log.gz 2024-09-18T00:42:29.208 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46863.log 2024-09-18T00:42:29.208 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.138794.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.138794.log.gz 2024-09-18T00:42:29.208 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148943.log 2024-09-18T00:42:29.208 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.46863.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100251.log 2024-09-18T00:42:29.208 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87703.log 2024-09-18T00:42:29.208 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100251.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.92271.log.gz 2024-09-18T00:42:29.208 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122636.log 2024-09-18T00:42:29.208 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100458.log 2024-09-18T00:42:29.208 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110722.log 2024-09-18T00:42:29.208 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147833.log 2024-09-18T00:42:29.208 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141542.log 2024-09-18T00:42:29.208 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147833.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42532.log.gz 2024-09-18T00:42:29.208 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56210.log 2024-09-18T00:42:29.209 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142035.log 2024-09-18T00:42:29.209 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.56210.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.147833.log.gz 2024-09-18T00:42:29.209 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107160.log 2024-09-18T00:42:29.209 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142035.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86753.log 2024-09-18T00:42:29.209 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82209.log.gz 2024-09-18T00:42:29.209 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107160.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113117.log.gz 2024-09-18T00:42:29.209 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54398.log 2024-09-18T00:42:29.209 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86753.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140313.log 2024-09-18T00:42:29.209 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66420.log 2024-09-18T00:42:29.209 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146742.log 2024-09-18T00:42:29.209 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1%/var/log/ceph/cluster1-client.mirror.66420.log: -- replaced with /var/log/ceph/cluster2-client.mirror.82437.log.gz 2024-09-18T00:42:29.209 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71757.log 2024-09-18T00:42:29.209 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101510.log 2024-09-18T00:42:29.210 INFO:teuthology.orchestra.run.smithi192.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.admin.59210.log.gz 2024-09-18T00:42:29.210 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39422.log 2024-09-18T00:42:29.210 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135325.log 2024-09-18T00:42:29.210 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39422.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95898.log 2024-09-18T00:42:29.210 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111648.log 2024-09-18T00:42:29.210 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138740.log 2024-09-18T00:42:29.210 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134045.log 2024-09-18T00:42:29.210 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111011.log.gz 2024-09-18T00:42:29.210 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51879.log 2024-09-18T00:42:29.210 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39402.log 2024-09-18T00:42:29.210 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51879.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111864.log 2024-09-18T00:42:29.210 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41692.log.gz 2024-09-18T00:42:29.210 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34609.log 2024-09-18T00:42:29.211 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111864.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41693.log.gz 2024-09-18T00:42:29.211 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78375.log 2024-09-18T00:42:29.211 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.34609.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34609.log.gz 2024-09-18T00:42:29.211 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130475.log 2024-09-18T00:42:29.211 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37848.log 2024-09-18T00:42:29.211 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.127221.log.gz 2024-09-18T00:42:29.211 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.130475.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146952.log 2024-09-18T00:42:29.211 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37848.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83431.log 2024-09-18T00:42:29.211 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153111.log 2024-09-18T00:42:29.211 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83431.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152193.log 2024-09-18T00:42:29.211 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.153111.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50955.log 2024-09-18T00:42:29.211 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.153111.log.gz 2024-09-18T00:42:29.211 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.152193.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146952.log.gz 2024-09-18T00:42:29.212 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128346.log 2024-09-18T00:42:29.212 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54441.log.gz 2024-09-18T00:42:29.212 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128209.log 2024-09-18T00:42:29.212 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115604.log 2024-09-18T00:42:29.212 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148314.log 2024-09-18T00:42:29.212 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84348.log 2024-09-18T00:42:29.212 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127627.log 2024-09-18T00:42:29.212 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.69648.log.gz 2024-09-18T00:42:29.212 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84348.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53767.log 2024-09-18T00:42:29.212 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142492.log 2024-09-18T00:42:29.212 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.148314.log.gz 2024-09-18T00:42:29.212 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53767.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88393.log 2024-09-18T00:42:29.213 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53484.log 2024-09-18T00:42:29.213 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88393.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42653.log 2024-09-18T00:42:29.213 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150804.log 2024-09-18T00:42:29.213 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149875.log 2024-09-18T00:42:29.213 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.150804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87415.log 2024-09-18T00:42:29.213 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149875.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123980.log.gz 2024-09-18T00:42:29.213 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138872.log 2024-09-18T00:42:29.213 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129492.log 2024-09-18T00:42:29.213 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.138872.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.138872.log.gz 2024-09-18T00:42:29.213 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135619.log 2024-09-18T00:42:29.213 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119891.log 2024-09-18T00:42:29.213 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135619.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137214.log 2024-09-18T00:42:29.213 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.119891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135263.log 2024-09-18T00:42:29.214 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137214.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35573.log 2024-09-18T00:42:29.214 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135263.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94742.log 2024-09-18T00:42:29.214 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.35573.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123727.log.gz 2024-09-18T00:42:29.214 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139042.log 2024-09-18T00:42:29.214 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.94742.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128906.log 2024-09-18T00:42:29.214 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139042.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105815.log 2024-09-18T00:42:29.214 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128906.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128906.log.gz 2024-09-18T00:42:29.214 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99282.log 2024-09-18T00:42:29.214 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.105815.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113872.log.gz 2024-09-18T00:42:29.214 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78733.log 2024-09-18T00:42:29.214 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99282.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123183.log 2024-09-18T00:42:29.214 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78733.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45596.log 2024-09-18T00:42:29.214 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109968.log 2024-09-18T00:42:29.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.45596.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79571.log 2024-09-18T00:42:29.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130652.log 2024-09-18T00:42:29.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79571.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.79571.log.gz 2024-09-18T00:42:29.215 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116358.log 2024-09-18T00:42:29.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72704.log 2024-09-18T00:42:29.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116358.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87163.log 2024-09-18T00:42:29.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72704.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67813.log 2024-09-18T00:42:29.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51836.log 2024-09-18T00:42:29.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123341.log 2024-09-18T00:42:29.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51836.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62658.log 2024-09-18T00:42:29.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146322.log 2024-09-18T00:42:29.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62658.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41272.log 2024-09-18T00:42:29.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130877.log 2024-09-18T00:42:29.216 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136312.log.gz 2024-09-18T00:42:29.216 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53559.log 2024-09-18T00:42:29.216 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49292.log 2024-09-18T00:42:29.216 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53559.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65094.log 2024-09-18T00:42:29.216 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49292.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77115.log 2024-09-18T00:42:29.216 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85303.log 2024-09-18T00:42:29.216 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47233.log 2024-09-18T00:42:29.216 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147163.log 2024-09-18T00:42:29.216 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.47233.log: gzip -5 90.0% --verbose -- -- replaced with /var/log/ceph/cluster1-client.mirror.37702.log.gz /var/log/ceph/cluster1-client.mirror.145083.log 2024-09-18T00:42:29.216 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:29.216 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142096.log 2024-09-18T00:42:29.216 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145083.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90743.log 2024-09-18T00:42:29.216 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58841.log 2024-09-18T00:42:29.217 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90743.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.147163.log.gz 2024-09-18T00:42:29.217 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98780.log 2024-09-18T00:42:29.217 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.58841.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63624.log.gz 2024-09-18T00:42:29.217 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139039.log 2024-09-18T00:42:29.217 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98780.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136964.log 2024-09-18T00:42:29.217 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148424.log 2024-09-18T00:42:29.217 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136964.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88455.log 2024-09-18T00:42:29.217 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44942.log 2024-09-18T00:42:29.217 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88455.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79898.log 2024-09-18T00:42:29.217 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124947.log 2024-09-18T00:42:29.217 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43333.log 2024-09-18T00:42:29.217 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124947.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84192.log 2024-09-18T00:42:29.217 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60365.log 2024-09-18T00:42:29.218 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84192.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70852.log 2024-09-18T00:42:29.218 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.60365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131154.log 2024-09-18T00:42:29.218 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.70852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116111.log 2024-09-18T00:42:29.218 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131154.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125789.log.gz 2024-09-18T00:42:29.218 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127643.log 2024-09-18T00:42:29.218 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116111.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75760.log 2024-09-18T00:42:29.218 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41064.log 2024-09-18T00:42:29.218 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130911.log 2024-09-18T00:42:29.218 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55138.log 2024-09-18T00:42:29.218 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.130911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143709.log 2024-09-18T00:42:29.218 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55138.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66420.log.gz 2024-09-18T00:42:29.218 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51796.log 2024-09-18T00:42:29.218 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100166.log 2024-09-18T00:42:29.219 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105742.log 2024-09-18T00:42:29.219 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100166.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105883.log 2024-09-18T00:42:29.219 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128043.log.gz 2024-09-18T00:42:29.219 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.105742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148844.log 2024-09-18T00:42:29.219 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.105883.log: gzip 87.3% -5 -- replaced with /var/log/ceph/cluster2-client.admin.128369.log.gz --verbose 2024-09-18T00:42:29.219 INFO:teuthology.orchestra.run.smithi192.stderr: -- /var/log/ceph/cluster1-client.mirror.112861.log 2024-09-18T00:42:29.219 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60143.log 2024-09-18T00:42:29.219 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112861.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80420.log 2024-09-18T00:42:29.219 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60143.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140952.log 2024-09-18T00:42:29.219 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.80420.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140216.log 2024-09-18T00:42:29.219 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.140952.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65430.log 2024-09-18T00:42:29.219 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.140216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131523.log 2024-09-18T00:42:29.219 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65829.log.gz 2024-09-18T00:42:29.220 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102267.log 2024-09-18T00:42:29.220 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.131523.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134864.log 2024-09-18T00:42:29.220 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143258.log 2024-09-18T00:42:29.220 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134864.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130715.log 2024-09-18T00:42:29.220 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143258.log: 79.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71606.log.gz 2024-09-18T00:42:29.220 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116092.log 2024-09-18T00:42:29.220 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71757.log.gz 2024-09-18T00:42:29.220 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146449.log 2024-09-18T00:42:29.220 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116092.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.71850.log.gz 2024-09-18T00:42:29.220 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125788.log 2024-09-18T00:42:29.220 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113701.log 2024-09-18T00:42:29.220 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125788.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148634.log 2024-09-18T00:42:29.220 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103491.log 2024-09-18T00:42:29.220 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148634.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115258.log 2024-09-18T00:42:29.221 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.103491.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122132.log 2024-09-18T00:42:29.221 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.35999.log.gz 2024-09-18T00:42:29.221 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35977.log.gz 2024-09-18T00:42:29.221 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85976.log 2024-09-18T00:42:29.221 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82229.log 2024-09-18T00:42:29.221 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137111.log 2024-09-18T00:42:29.221 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.82229.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64464.log 2024-09-18T00:42:29.221 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152663.log 2024-09-18T00:42:29.221 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136271.log 2024-09-18T00:42:29.221 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152663.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.152663.log.gz 2024-09-18T00:42:29.221 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68585.log 2024-09-18T00:42:29.221 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85932.log 2024-09-18T00:42:29.222 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68585.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86456.log 2024-09-18T00:42:29.222 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99660.log 2024-09-18T00:42:29.222 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40642.log.gz 2024-09-18T00:42:29.222 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64507.log 2024-09-18T00:42:29.222 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99660.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57140.log 2024-09-18T00:42:29.222 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42746.log 2024-09-18T00:42:29.222 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.57140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63310.log 2024-09-18T00:42:29.222 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42746.log: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47642.log.gz 2024-09-18T00:42:29.222 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114292.log 2024-09-18T00:42:29.222 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93181.log 2024-09-18T00:42:29.222 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114292.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143297.log 2024-09-18T00:42:29.222 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47835.log.gz 2024-09-18T00:42:29.222 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.93181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45299.log 2024-09-18T00:42:29.223 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143297.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145281.log 2024-09-18T00:42:29.223 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114378.log 2024-09-18T00:42:29.223 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145281.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65514.log 2024-09-18T00:42:29.223 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114378.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60018.log 2024-09-18T00:42:29.223 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145963.log.gz 2024-09-18T00:42:29.223 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143146.log 2024-09-18T00:42:29.223 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77115.log.gz 2024-09-18T00:42:29.223 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65537.log 2024-09-18T00:42:29.223 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.143146.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149114.log 2024-09-18T00:42:29.223 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74582.log 2024-09-18T00:42:29.223 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149114.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39004.log 2024-09-18T00:42:29.223 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74582.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129082.log 2024-09-18T00:42:29.223 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78205.log 2024-09-18T00:42:29.223 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67110.log.gz 2024-09-18T00:42:29.224 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73145.log.gz 2024-09-18T00:42:29.224 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133054.log 2024-09-18T00:42:29.224 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78205.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113952.log 2024-09-18T00:42:29.224 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133054.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47206.log 2024-09-18T00:42:29.224 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113952.log: gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-18T00:42:29.224 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.47206.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66271.log 2024-09-18T00:42:29.224 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42124.log 2024-09-18T00:42:29.224 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92241.log 2024-09-18T00:42:29.224 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138264.log 2024-09-18T00:42:29.224 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46508.log 2024-09-18T00:42:29.224 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138264.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151761.log 2024-09-18T00:42:29.224 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46508.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46508.log.gz 2024-09-18T00:42:29.225 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47259.log 2024-09-18T00:42:29.225 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.151761.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42956.log 2024-09-18T00:42:29.225 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.143258.log.gz 2024-09-18T00:42:29.225 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.47259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110512.log 2024-09-18T00:42:29.225 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42956.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103741.log 2024-09-18T00:42:29.225 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84014.log 2024-09-18T00:42:29.225 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103741.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68855.log 2024-09-18T00:42:29.225 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54695.log 2024-09-18T00:42:29.225 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.68855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103893.log 2024-09-18T00:42:29.225 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54695.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137214.log.gz 2024-09-18T00:42:29.225 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116766.log 2024-09-18T00:42:29.249 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103893.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70671.log 2024-09-18T00:42:29.249 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.116766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88542.log 2024-09-18T00:42:29.250 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111142.log 2024-09-18T00:42:29.250 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88542.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45596.log.gz 2024-09-18T00:42:29.250 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92295.log 2024-09-18T00:42:29.250 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111142.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87398.log 2024-09-18T00:42:29.250 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92295.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43372.log 2024-09-18T00:42:29.250 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87398.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153453.log 2024-09-18T00:42:29.250 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111017.log 2024-09-18T00:42:29.250 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.153453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82647.log 2024-09-18T00:42:29.250 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.153453.log.gz 2024-09-18T00:42:29.250 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111017.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125660.log.gz 2024-09-18T00:42:29.250 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137953.log 2024-09-18T00:42:29.250 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.82647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143431.log 2024-09-18T00:42:29.250 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38448.log 2024-09-18T00:42:29.251 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138873.log 2024-09-18T00:42:29.251 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118869.log 2024-09-18T00:42:29.251 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120066.log 2024-09-18T00:42:29.251 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118869.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148733.log 2024-09-18T00:42:29.251 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100166.log.gz 2024-09-18T00:42:29.251 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120066.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.151575.log 2024-09-18T00:42:29.251 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148733.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79322.log 2024-09-18T00:42:29.251 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.151575.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.100207.log.gz 2024-09-18T00:42:29.251 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100587.log 2024-09-18T00:42:29.251 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86309.log 2024-09-18T00:42:29.251 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118513.log 2024-09-18T00:42:29.251 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.100251.log.gz 2024-09-18T00:42:29.252 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93944.log 2024-09-18T00:42:29.252 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118513.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30352.log 2024-09-18T00:42:29.252 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.93944.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87208.log.gz 2024-09-18T00:42:29.252 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47164.log 2024-09-18T00:42:29.252 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.30352.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30352.log.gz 2024-09-18T00:42:29.252 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76065.log 2024-09-18T00:42:29.252 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.47164.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88583.log 2024-09-18T00:42:29.252 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126756.log 2024-09-18T00:42:29.252 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88583.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.86890.log.gz 2024-09-18T00:42:29.252 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101908.log 2024-09-18T00:42:29.252 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87659.log 2024-09-18T00:42:29.253 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74734.log 2024-09-18T00:42:29.253 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87659.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45993.log 2024-09-18T00:42:29.253 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.36881.log.gz 2024-09-18T00:42:29.253 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74734.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39001.log 2024-09-18T00:42:29.253 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51628.log 2024-09-18T00:42:29.253 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106271.log 2024-09-18T00:42:29.253 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51628.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50746.log 2024-09-18T00:42:29.253 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.106271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118155.log 2024-09-18T00:42:29.253 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123517.log 2024-09-18T00:42:29.253 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.118155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148422.log 2024-09-18T00:42:29.253 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123517.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.88455.log.gz 2024-09-18T00:42:29.253 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88061.log 2024-09-18T00:42:29.253 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148422.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37995.log 2024-09-18T00:42:29.254 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75887.log 2024-09-18T00:42:29.254 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77894.log 2024-09-18T00:42:29.254 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114161.log 2024-09-18T00:42:29.254 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111641.log 2024-09-18T00:42:29.254 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114161.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97686.log 2024-09-18T00:42:29.254 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64844.log 2024-09-18T00:42:29.254 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149573.log 2024-09-18T00:42:29.254 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150669.log 2024-09-18T00:42:29.254 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84059.log 2024-09-18T00:42:29.254 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83368.log 2024-09-18T00:42:29.254 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135199.log 2024-09-18T00:42:29.254 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.140952.log.gz 2024-09-18T00:42:29.255 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83368.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75089.log 2024-09-18T00:42:29.255 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.84348.log.gz 2024-09-18T00:42:29.255 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107617.log 2024-09-18T00:42:29.255 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115218.log 2024-09-18T00:42:29.255 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79045.log 2024-09-18T00:42:29.255 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.115218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56345.log 2024-09-18T00:42:29.255 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35951.log 2024-09-18T00:42:29.255 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56345.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98047.log 2024-09-18T00:42:29.255 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.78375.log.gz 2024-09-18T00:42:29.255 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.35951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147644.log 2024-09-18T00:42:29.255 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98047.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117980.log 2024-09-18T00:42:29.255 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113954.log 2024-09-18T00:42:29.255 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.117980.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59733.log 2024-09-18T00:42:29.256 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146382.log 2024-09-18T00:42:29.256 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59733.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.100458.log.gz 2024-09-18T00:42:29.256 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119003.log 2024-09-18T00:42:29.256 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.147644.log.gz 2024-09-18T00:42:29.256 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70215.log 2024-09-18T00:42:29.256 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136374.log 2024-09-18T00:42:29.256 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82364.log 2024-09-18T00:42:29.256 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148633.log 2024-09-18T00:42:29.256 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.82364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112691.log 2024-09-18T00:42:29.256 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148633.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73513.log 2024-09-18T00:42:29.256 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.99913.log.gz 2024-09-18T00:42:29.256 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54527.log 2024-09-18T00:42:29.256 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62576.log 2024-09-18T00:42:29.256 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54527.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93248.log 2024-09-18T00:42:29.257 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62576.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105701.log 2024-09-18T00:42:29.257 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93248.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74668.log 2024-09-18T00:42:29.257 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.105701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38578.log 2024-09-18T00:42:29.257 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113871.log 2024-09-18T00:42:29.257 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38578.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59803.log 2024-09-18T00:42:29.257 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113871.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147116.log 2024-09-18T00:42:29.257 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59803.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115943.log 2024-09-18T00:42:29.257 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147116.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134170.log 2024-09-18T00:42:29.257 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.115943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98116.log 2024-09-18T00:42:29.257 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134170.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42536.log 2024-09-18T00:42:29.257 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98116.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109599.log.gz 2024-09-18T00:42:29.257 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109546.log.gz 2024-09-18T00:42:29.257 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34693.log 2024-09-18T00:42:29.258 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39215.log 2024-09-18T00:42:29.258 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.34693.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34693.log.gz 2024-09-18T00:42:29.266 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39834.log 2024-09-18T00:42:29.266 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.105883.log.gz 2024-09-18T00:42:29.266 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141355.log 2024-09-18T00:42:29.267 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39834.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.150830.log.gz 2024-09-18T00:42:29.267 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50514.log 2024-09-18T00:42:29.267 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.152828.log 2024-09-18T00:42:29.267 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82886.log 2024-09-18T00:42:29.267 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.152828.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125534.log 2024-09-18T00:42:29.267 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.152828.log.gz 2024-09-18T00:42:29.267 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.82886.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.151575.log.gz 2024-09-18T00:42:29.267 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70811.log 2024-09-18T00:42:29.267 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133540.log 2024-09-18T00:42:29.267 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70811.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.151761.log.gz 2024-09-18T00:42:29.267 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123477.log 2024-09-18T00:42:29.267 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.151894.log.gz 2024-09-18T00:42:29.267 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54567.log 2024-09-18T00:42:29.268 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115020.log 2024-09-18T00:42:29.268 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54567.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.152193.log.gz 2024-09-18T00:42:29.268 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133918.log 2024-09-18T00:42:29.268 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115020.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.115020.log.gz 2024-09-18T00:42:29.268 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93585.log 2024-09-18T00:42:29.268 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5%/var/log/ceph/cluster1-client.mirror.133918.log: -- replaced with /var/log/ceph/cluster2-client.mirror.152332.log.gz 2024-09-18T00:42:29.268 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42533.log 2024-09-18T00:42:29.268 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93585.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.93585.log.gz 2024-09-18T00:42:29.268 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76423.log 2024-09-18T00:42:29.268 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145016.log 2024-09-18T00:42:29.268 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112698.log 2024-09-18T00:42:29.268 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145016.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45187.log 2024-09-18T00:42:29.269 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112698.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139873.log 2024-09-18T00:42:29.269 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.45187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90180.log 2024-09-18T00:42:29.269 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54946.log 2024-09-18T00:42:29.269 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90180.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121799.log 2024-09-18T00:42:29.269 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120066.log.gz 2024-09-18T00:42:29.269 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54946.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143669.log 2024-09-18T00:42:29.269 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124568.log 2024-09-18T00:42:29.269 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56102.log 2024-09-18T00:42:29.269 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124568.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48216.log 2024-09-18T00:42:29.269 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43372.log.gz 2024-09-18T00:42:29.269 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56102.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.56102.log.gz 2024-09-18T00:42:29.269 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68330.log 2024-09-18T00:42:29.270 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101698.log 2024-09-18T00:42:29.270 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110177.log 2024-09-18T00:42:29.277 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.58613.log.gz 2024-09-18T00:42:29.277 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63312.log 2024-09-18T00:42:29.278 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74054.log 2024-09-18T00:42:29.278 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.123098.log.gz 2024-09-18T00:42:29.278 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86583.log 2024-09-18T00:42:29.278 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.82364.log.gz 2024-09-18T00:42:29.278 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74054.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.82384.log.gz 2024-09-18T00:42:29.278 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74458.log 2024-09-18T00:42:29.278 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148212.log 2024-09-18T00:42:29.278 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74458.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52676.log.gz 2024-09-18T00:42:29.278 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133078.log 2024-09-18T00:42:29.278 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57012.log 2024-09-18T00:42:29.278 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133078.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121670.log 2024-09-18T00:42:29.278 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57012.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70811.log.gz 2024-09-18T00:42:29.279 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135012.log 2024-09-18T00:42:29.279 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77895.log 2024-09-18T00:42:29.279 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78584.log 2024-09-18T00:42:29.279 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93678.log 2024-09-18T00:42:29.279 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106339.log 2024-09-18T00:42:29.279 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93678.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.93678.log.gz 2024-09-18T00:42:29.279 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44640.log 2024-09-18T00:42:29.279 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.106339.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.149426.log.gz 2024-09-18T00:42:29.279 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88060.log 2024-09-18T00:42:29.279 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149052.log 2024-09-18T00:42:29.279 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88060.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105681.log 2024-09-18T00:42:29.279 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149052.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.82886.log.gz 2024-09-18T00:42:29.279 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75509.log 2024-09-18T00:42:29.280 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.105681.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106178.log 2024-09-18T00:42:29.283 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79703.log 2024-09-18T00:42:29.283 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145778.log 2024-09-18T00:42:29.283 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79703.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.79703.log.gz 2024-09-18T00:42:29.283 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113536.log 2024-09-18T00:42:29.283 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145778.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53290.log 2024-09-18T00:42:29.284 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.115943.log.gz 2024-09-18T00:42:29.284 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59407.log 2024-09-18T00:42:29.284 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143359.log 2024-09-18T00:42:29.284 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99200.log 2024-09-18T00:42:29.284 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.143359.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51249.log 2024-09-18T00:42:29.284 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99200.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45992.log 2024-09-18T00:42:29.284 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51249.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42653.log.gz 2024-09-18T00:42:29.284 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93477.log 2024-09-18T00:42:29.284 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125999.log 2024-09-18T00:42:29.286 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93247.log 2024-09-18T00:42:29.286 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.116092.log.gz 2024-09-18T00:42:29.287 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86518.log 2024-09-18T00:42:29.287 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93247.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.116224.log.gz 2024-09-18T00:42:29.287 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91095.log 2024-09-18T00:42:29.287 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112232.log 2024-09-18T00:42:29.287 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91095.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86142.log 2024-09-18T00:42:29.287 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.142096.log.gz 2024-09-18T00:42:29.287 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86101.log 2024-09-18T00:42:29.288 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86142.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114731.log 2024-09-18T00:42:29.289 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55617.log 2024-09-18T00:42:29.289 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114731.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87248.log 2024-09-18T00:42:29.289 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55238.log 2024-09-18T00:42:29.290 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87248.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83978.log 2024-09-18T00:42:29.290 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59028.log 2024-09-18T00:42:29.290 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42746.log.gz 2024-09-18T00:42:29.290 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150102.log 2024-09-18T00:42:29.291 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.59028.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80503.log 2024-09-18T00:42:29.291 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150102.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130015.log 2024-09-18T00:42:29.292 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.151574.log 2024-09-18T00:42:29.292 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129518.log 2024-09-18T00:42:29.293 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113952.log.gz 2024-09-18T00:42:29.293 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.151574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91102.log 2024-09-18T00:42:29.294 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153514.log 2024-09-18T00:42:29.294 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91102.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135282.log 2024-09-18T00:42:29.295 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.153514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149684.log 2024-09-18T00:42:29.295 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.153514.log.gz 2024-09-18T00:42:29.295 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135282.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65960.log 2024-09-18T00:42:29.295 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149684.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47528.log 2024-09-18T00:42:29.296 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65960.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43584.log 2024-09-18T00:42:29.296 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47528.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70329.log 2024-09-18T00:42:29.297 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117954.log 2024-09-18T00:42:29.298 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56098.log 2024-09-18T00:42:29.298 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.117954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106312.log 2024-09-18T00:42:29.299 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.56098.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112481.log.gz 2024-09-18T00:42:29.299 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77534.log 2024-09-18T00:42:29.299 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.106312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108851.log 2024-09-18T00:42:29.300 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143305.log 2024-09-18T00:42:29.302 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90853.log 2024-09-18T00:42:29.302 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62409.log 2024-09-18T00:42:29.303 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90853.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114292.log.gz 2024-09-18T00:42:29.303 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103401.log 2024-09-18T00:42:29.303 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76632.log 2024-09-18T00:42:29.303 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103401.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.103401.log.gz 2024-09-18T00:42:29.303 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106023.log 2024-09-18T00:42:29.305 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76632.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146953.log 2024-09-18T00:42:29.305 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.106023.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97353.log 2024-09-18T00:42:29.305 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146953.log: 89.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57588.log 2024-09-18T00:42:29.305 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.134970.log.gz 2024-09-18T00:42:29.305 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114331.log 2024-09-18T00:42:29.306 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.57588.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150438.log 2024-09-18T00:42:29.306 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.146953.log.gz 2024-09-18T00:42:29.306 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70282.log 2024-09-18T00:42:29.306 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150438.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.150438.log.gz 2024-09-18T00:42:29.307 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123099.log 2024-09-18T00:42:29.307 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.70282.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118668.log 2024-09-18T00:42:29.308 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123099.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.84059.log.gz 2024-09-18T00:42:29.308 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145585.log 2024-09-18T00:42:29.308 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78920.log 2024-09-18T00:42:29.309 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145585.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63944.log 2024-09-18T00:42:29.310 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112066.log 2024-09-18T00:42:29.310 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135389.log 2024-09-18T00:42:29.310 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96188.log 2024-09-18T00:42:29.311 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39863.log 2024-09-18T00:42:29.311 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82830.log 2024-09-18T00:42:29.312 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39863.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97814.log 2024-09-18T00:42:29.313 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.82830.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120087.log 2024-09-18T00:42:29.313 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74624.log 2024-09-18T00:42:29.314 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120087.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110802.log 2024-09-18T00:42:29.314 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74624.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.142165.log.gz 2024-09-18T00:42:29.314 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118410.log 2024-09-18T00:42:29.315 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124486.log 2024-09-18T00:42:29.315 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.118410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101088.log 2024-09-18T00:42:29.315 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124486.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97704.log 2024-09-18T00:42:29.316 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107569.log 2024-09-18T00:42:29.316 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97704.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124781.log 2024-09-18T00:42:29.317 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.107569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47144.log 2024-09-18T00:42:29.317 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78143.log 2024-09-18T00:42:29.317 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.47144.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103327.log.gz 2024-09-18T00:42:29.318 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48131.log 2024-09-18T00:42:29.318 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60938.log 2024-09-18T00:42:29.318 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.48131.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103491.log.gz 2024-09-18T00:42:29.318 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122092.log 2024-09-18T00:42:29.319 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107090.log 2024-09-18T00:42:29.319 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37404.log 2024-09-18T00:42:29.320 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107090.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141522.log 2024-09-18T00:42:29.320 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37404.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88270.log 2024-09-18T00:42:29.321 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141522.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50998.log 2024-09-18T00:42:29.321 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105722.log 2024-09-18T00:42:29.322 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50998.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76004.log 2024-09-18T00:42:29.322 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.48483.log.gz 2024-09-18T00:42:29.323 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.105722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90432.log 2024-09-18T00:42:29.323 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76004.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.48789.log.gz 2024-09-18T00:42:29.323 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117345.log 2024-09-18T00:42:29.324 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48395.log 2024-09-18T00:42:29.324 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.117345.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.117345.log.gz 2024-09-18T00:42:29.324 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119458.log 2024-09-18T00:42:29.324 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48979.log.gz 2024-09-18T00:42:29.325 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48395.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121120.log 2024-09-18T00:42:29.325 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59801.log 2024-09-18T00:42:29.326 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54298.log 2024-09-18T00:42:29.326 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143121.log 2024-09-18T00:42:29.327 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54298.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93249.log 2024-09-18T00:42:29.327 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122913.log 2024-09-18T00:42:29.327 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.136271.log.gz 2024-09-18T00:42:29.327 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93249.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150838.log 2024-09-18T00:42:29.328 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122913.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143305.log.gz 2024-09-18T00:42:29.329 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98022.log 2024-09-18T00:42:29.329 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150838.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127718.log 2024-09-18T00:42:29.329 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98022.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45851.log 2024-09-18T00:42:29.330 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.127718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147893.log 2024-09-18T00:42:29.330 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.45851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80823.log 2024-09-18T00:42:29.331 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147893.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93810.log 2024-09-18T00:42:29.331 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.70329.log.gz 2024-09-18T00:42:29.331 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141791.log 2024-09-18T00:42:29.332 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.93810.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67279.log 2024-09-18T00:42:29.333 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.141791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67488.log 2024-09-18T00:42:29.333 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67279.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89799.log 2024-09-18T00:42:29.334 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74054.log.gz 2024-09-18T00:42:29.334 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67488.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140321.log 2024-09-18T00:42:29.335 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77955.log 2024-09-18T00:42:29.335 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83093.log 2024-09-18T00:42:29.335 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153225.log 2024-09-18T00:42:29.336 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122554.log 2024-09-18T00:42:29.336 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.153225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64338.log 2024-09-18T00:42:29.336 INFO:teuthology.orchestra.run.smithi192.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.153225.log.gz 2024-09-18T00:42:29.337 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.136189.log.gz/var/log/ceph/cluster1-client.mirror.122554.log: 2024-09-18T00:42:29.337 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126855.log 2024-09-18T00:42:29.338 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64338.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98362.log 2024-09-18T00:42:29.338 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127842.log 2024-09-18T00:42:29.338 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59539.log 2024-09-18T00:42:29.339 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.127842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91141.log 2024-09-18T00:42:29.339 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145062.log 2024-09-18T00:42:29.340 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91141.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.145281.log.gz 2024-09-18T00:42:29.340 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128840.log 2024-09-18T00:42:29.340 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58707.log 2024-09-18T00:42:29.341 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103897.log 2024-09-18T00:42:29.342 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.58707.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64906.log 2024-09-18T00:42:29.342 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103897.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90616.log 2024-09-18T00:42:29.343 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99096.log 2024-09-18T00:42:29.343 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123624.log 2024-09-18T00:42:29.344 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42912.log 2024-09-18T00:42:29.344 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110301.log 2024-09-18T00:42:29.345 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107806.log 2024-09-18T00:42:29.345 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146532.log 2024-09-18T00:42:29.345 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107806.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66522.log 2024-09-18T00:42:29.346 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92064.log 2024-09-18T00:42:29.346 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66522.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65177.log 2024-09-18T00:42:29.348 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138840.log 2024-09-18T00:42:29.348 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125999.log.gz 2024-09-18T00:42:29.348 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65177.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146532.log.gz 2024-09-18T00:42:29.348 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152318.log 2024-09-18T00:42:29.348 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.138840.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.138840.log.gz 2024-09-18T00:42:29.348 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112065.log 2024-09-18T00:42:29.349 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.152318.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149054.log 2024-09-18T00:42:29.349 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90347.log 2024-09-18T00:42:29.350 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149054.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85861.log 2024-09-18T00:42:29.350 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147536.log 2024-09-18T00:42:29.351 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85861.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135325.log.gz 2024-09-18T00:42:29.351 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37282.log 2024-09-18T00:42:29.351 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147536.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118342.log 2024-09-18T00:42:29.352 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37282.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80367.log 2024-09-18T00:42:29.352 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.118342.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41652.log 2024-09-18T00:42:29.352 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.80367.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133918.log.gz 2024-09-18T00:42:29.353 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99912.log 2024-09-18T00:42:29.353 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54692.log 2024-09-18T00:42:29.354 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99912.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80975.log 2024-09-18T00:42:29.354 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48715.log 2024-09-18T00:42:29.355 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37143.log 2024-09-18T00:42:29.355 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44889.log 2024-09-18T00:42:29.356 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78143.log.gz 2024-09-18T00:42:29.356 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97919.log 2024-09-18T00:42:29.356 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44889.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136249.log 2024-09-18T00:42:29.357 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146783.log 2024-09-18T00:42:29.357 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136249.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124720.log 2024-09-18T00:42:29.358 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146783.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113537.log 2024-09-18T00:42:29.358 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136484.log 2024-09-18T00:42:29.359 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.146783.log.gz 2024-09-18T00:42:29.359 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74960.log 2024-09-18T00:42:29.359 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136484.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148274.log 2024-09-18T00:42:29.360 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74960.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88541.log 2024-09-18T00:42:29.360 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87248.log.gz 2024-09-18T00:42:29.360 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153855.log 2024-09-18T00:42:29.361 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126564.log 2024-09-18T00:42:29.361 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.153855.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.153855.log.gz 2024-09-18T00:42:29.362 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71208.log 2024-09-18T00:42:29.362 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.148274.log.gz 2024-09-18T00:42:29.362 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74120.log 2024-09-18T00:42:29.363 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44702.log 2024-09-18T00:42:29.363 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89590.log 2024-09-18T00:42:29.364 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67883.log 2024-09-18T00:42:29.365 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89590.log: gzip 89.8% -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101468.log 2024-09-18T00:42:29.365 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.78584.log.gz 2024-09-18T00:42:29.365 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88061.log.gz 2024-09-18T00:42:29.365 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97728.log 2024-09-18T00:42:29.366 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101468.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139603.log 2024-09-18T00:42:29.366 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91342.log 2024-09-18T00:42:29.367 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40487.log 2024-09-18T00:42:29.367 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91342.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91342.log.gz 2024-09-18T00:42:29.367 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88875.log 2024-09-18T00:42:29.367 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.87659.log.gz 2024-09-18T00:42:29.368 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53390.log 2024-09-18T00:42:29.368 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132745.log 2024-09-18T00:42:29.369 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151172.log 2024-09-18T00:42:29.369 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.132745.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127218.log 2024-09-18T00:42:29.371 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94017.log 2024-09-18T00:42:29.371 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102518.log 2024-09-18T00:42:29.371 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.94017.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.98362.log.gz 2024-09-18T00:42:29.371 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107028.log 2024-09-18T00:42:29.371 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119828.log 2024-09-18T00:42:29.372 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138631.log 2024-09-18T00:42:29.372 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119828.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135073.log 2024-09-18T00:42:29.372 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.138631.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.138631.log.gz 2024-09-18T00:42:29.373 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82457.log 2024-09-18T00:42:29.373 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74582.log.gz 2024-09-18T00:42:29.373 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118823.log 2024-09-18T00:42:29.374 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.82457.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77055.log 2024-09-18T00:42:29.374 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.118823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105655.log 2024-09-18T00:42:29.375 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30485.log 2024-09-18T00:42:29.376 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.105655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34651.log 2024-09-18T00:42:29.376 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.30485.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120046.log 2024-09-18T00:42:29.376 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30485.log.gz 2024-09-18T00:42:29.376 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.34651.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34651.log.gz 2024-09-18T00:42:29.376 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62700.log 2024-09-18T00:42:29.377 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120046.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47992.log 2024-09-18T00:42:29.377 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62283.log 2024-09-18T00:42:29.378 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.47992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53962.log 2024-09-18T00:42:29.378 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56904.log 2024-09-18T00:42:29.378 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53962.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76904.log 2024-09-18T00:42:29.379 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.56904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126418.log 2024-09-18T00:42:29.380 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112192.log 2024-09-18T00:42:29.380 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126418.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147055.log 2024-09-18T00:42:29.380 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112192.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103492.log 2024-09-18T00:42:29.381 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147055.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142033.log 2024-09-18T00:42:29.381 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.103492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119388.log 2024-09-18T00:42:29.382 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.147055.log.gz 2024-09-18T00:42:29.382 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145272.log 2024-09-18T00:42:29.383 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119388.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116099.log 2024-09-18T00:42:29.383 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53056.log 2024-09-18T00:42:29.383 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116099.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44571.log 2024-09-18T00:42:29.384 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53056.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116753.log 2024-09-18T00:42:29.385 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113912.log 2024-09-18T00:42:29.385 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138163.log.gz 2024-09-18T00:42:29.385 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.116753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89881.log 2024-09-18T00:42:29.386 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55406.log 2024-09-18T00:42:29.386 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134635.log 2024-09-18T00:42:29.387 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55406.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79535.log 2024-09-18T00:42:29.387 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134635.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103065.log.gz 2024-09-18T00:42:29.387 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62913.log 2024-09-18T00:42:29.388 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110598.log 2024-09-18T00:42:29.388 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.79535.log.gz 2024-09-18T00:42:29.388 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62913.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125096.log 2024-09-18T00:42:29.389 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110598.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50998.log.gz 2024-09-18T00:42:29.389 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115153.log 2024-09-18T00:42:29.389 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40319.log 2024-09-18T00:42:29.395 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.115153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78526.log 2024-09-18T00:42:29.395 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77156.log 2024-09-18T00:42:29.395 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146989.log 2024-09-18T00:42:29.395 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70872.log 2024-09-18T00:42:29.395 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50164.log 2024-09-18T00:42:29.395 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70872.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87102.log 2024-09-18T00:42:29.396 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50164.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62470.log 2024-09-18T00:42:29.396 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87102.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128893.log 2024-09-18T00:42:29.396 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.146989.log.gz 2024-09-18T00:42:29.396 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99200.log.gz 2024-09-18T00:42:29.396 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62470.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112022.log 2024-09-18T00:42:29.396 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128893.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128893.log.gz 2024-09-18T00:42:29.396 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86228.log 2024-09-18T00:42:29.397 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82156.log 2024-09-18T00:42:29.398 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107769.log 2024-09-18T00:42:29.398 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.82156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130869.log 2024-09-18T00:42:29.398 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136902.log 2024-09-18T00:42:29.398 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130869.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66774.log 2024-09-18T00:42:29.399 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118249.log 2024-09-18T00:42:29.399 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124822.log.gz 2024-09-18T00:42:29.399 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75632.log 2024-09-18T00:42:29.399 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118249.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111014.log 2024-09-18T00:42:29.400 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75632.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115066.log 2024-09-18T00:42:29.400 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97289.log 2024-09-18T00:42:29.400 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.115066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35913.log 2024-09-18T00:42:29.401 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38999.log 2024-09-18T00:42:29.401 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.35913.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79901.log 2024-09-18T00:42:29.402 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37281.log 2024-09-18T00:42:29.402 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79901.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78037.log 2024-09-18T00:42:29.403 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37281.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42322.log 2024-09-18T00:42:29.403 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78037.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54020.log 2024-09-18T00:42:29.404 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88013.log 2024-09-18T00:42:29.405 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151358.log 2024-09-18T00:42:29.405 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142735.log 2024-09-18T00:42:29.405 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151358.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150472.log 2024-09-18T00:42:29.406 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146785.log 2024-09-18T00:42:29.407 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150472.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.150472.log.gz 2024-09-18T00:42:29.407 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.51585.log.gz -5 --verbose 2024-09-18T00:42:29.407 INFO:teuthology.orchestra.run.smithi192.stderr: -- /var/log/ceph/cluster1-client.mirror.39800.log 2024-09-18T00:42:29.407 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.81700.log 2024-09-18T00:42:29.408 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39800.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153359.log 2024-09-18T00:42:29.408 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.146785.log.gz 2024-09-18T00:42:29.408 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.81700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43793.log 2024-09-18T00:42:29.409 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.153359.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79725.log 2024-09-18T00:42:29.409 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97646.log 2024-09-18T00:42:29.409 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79725.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124379.log 2024-09-18T00:42:29.410 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97646.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46916.log 2024-09-18T00:42:29.411 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59277.log 2024-09-18T00:42:29.411 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101698.log.gz 2024-09-18T00:42:29.411 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.46916.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110551.log 2024-09-18T00:42:29.412 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59277.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88918.log 2024-09-18T00:42:29.412 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110551.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152188.log 2024-09-18T00:42:29.413 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52785.log 2024-09-18T00:42:29.413 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81908.log 2024-09-18T00:42:29.414 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136691.log 2024-09-18T00:42:29.414 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.81908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91518.log 2024-09-18T00:42:29.415 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60627.log 2024-09-18T00:42:29.415 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91518.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.143669.log.gz 2024-09-18T00:42:29.415 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.153359.log.gz 2024-09-18T00:42:29.416 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88813.log 2024-09-18T00:42:29.416 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82022.log 2024-09-18T00:42:29.417 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84214.log 2024-09-18T00:42:29.417 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.82022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150338.log 2024-09-18T00:42:29.418 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84214.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58093.log 2024-09-18T00:42:29.418 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150338.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70302.log 2024-09-18T00:42:29.419 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.58093.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.58093.log.gz 2024-09-18T00:42:29.419 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53960.log 2024-09-18T00:42:29.419 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70302.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39404.log 2024-09-18T00:42:29.420 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53960.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78836.log 2024-09-18T00:42:29.420 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39404.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114161.log.gz 2024-09-18T00:42:29.420 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152404.log 2024-09-18T00:42:29.421 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78836.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98420.log 2024-09-18T00:42:29.422 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152404.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64425.log 2024-09-18T00:42:29.422 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98420.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131758.log 2024-09-18T00:42:29.423 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64425.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140326.log 2024-09-18T00:42:29.423 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54881.log 2024-09-18T00:42:29.423 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116365.log 2024-09-18T00:42:29.424 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63269.log 2024-09-18T00:42:29.425 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.53265.log.gz 2024-09-18T00:42:29.425 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96076.log 2024-09-18T00:42:29.425 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126208.log 2024-09-18T00:42:29.426 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96076.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.150751.log 2024-09-18T00:42:29.426 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151253.log 2024-09-18T00:42:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.150751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53877.log 2024-09-18T00:42:29.427 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59865.log.gz 2024-09-18T00:42:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82960.log 2024-09-18T00:42:29.428 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53877.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92938.log 2024-09-18T00:42:29.428 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.139603.log.gz 2024-09-18T00:42:29.428 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.82960.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66986.log 2024-09-18T00:42:29.429 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122133.log 2024-09-18T00:42:29.429 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66986.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49341.log 2024-09-18T00:42:29.430 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122133.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130062.log 2024-09-18T00:42:29.430 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79236.log 2024-09-18T00:42:29.431 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140747.log 2024-09-18T00:42:29.432 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79236.log: /var/log/ceph/cluster1-client.admin.130062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150856.log 2024-09-18T00:42:29.432 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54230.log.gz 2024-09-18T00:42:29.432 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140747.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140747.log.gz 2024-09-18T00:42:29.432 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111352.log 2024-09-18T00:42:29.433 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90932.log 2024-09-18T00:42:29.433 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42747.log.gz 2024-09-18T00:42:29.433 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147222.log 2024-09-18T00:42:29.434 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90932.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56914.log 2024-09-18T00:42:29.435 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108386.log 2024-09-18T00:42:29.435 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123099.log.gz 2024-09-18T00:42:29.435 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.56914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81649.log 2024-09-18T00:42:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108386.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.147222.log.gz 2024-09-18T00:42:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148632.log 2024-09-18T00:42:29.437 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.119828.log.gz 2024-09-18T00:42:29.437 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81649.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81649.log.gz 2024-09-18T00:42:29.437 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39969.log 2024-09-18T00:42:29.437 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148632.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81058.log 2024-09-18T00:42:29.438 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39969.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83823.log 2024-09-18T00:42:29.438 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115941.log 2024-09-18T00:42:29.439 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101699.log 2024-09-18T00:42:29.439 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.115941.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58067.log 2024-09-18T00:42:29.440 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101699.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68084.log 2024-09-18T00:42:29.440 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52028.log.gz 2024-09-18T00:42:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.58067.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.58067.log.gz 2024-09-18T00:42:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49270.log 2024-09-18T00:42:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101636.log 2024-09-18T00:42:29.442 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49270.log: /var/log/ceph/cluster1-client.mirror.101636.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70925.log 2024-09-18T00:42:29.443 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.116099.log.gz 2024-09-18T00:42:29.443 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91386.log 2024-09-18T00:42:29.443 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113537.log.gz 2024-09-18T00:42:29.443 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113536.log.gz 2024-09-18T00:42:29.444 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70925.log: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.141791.log.gz 2024-09-18T00:42:29.444 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131369.log 2024-09-18T00:42:29.444 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91386.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112021.log 2024-09-18T00:42:29.445 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.131369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99097.log 2024-09-18T00:42:29.445 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112021.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35488.log 2024-09-18T00:42:29.446 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99097.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135494.log 2024-09-18T00:42:29.446 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.35488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44712.log 2024-09-18T00:42:29.447 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135494.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79005.log 2024-09-18T00:42:29.447 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90222.log 2024-09-18T00:42:29.448 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79005.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63585.log 2024-09-18T00:42:29.448 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71794.log 2024-09-18T00:42:29.449 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63585.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52006.log 2024-09-18T00:42:29.450 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71794.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139361.log 2024-09-18T00:42:29.450 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52006.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124151.log 2024-09-18T00:42:29.451 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139361.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96076.log.gz 2024-09-18T00:42:29.451 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42072.log 2024-09-18T00:42:29.451 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124151.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45259.log 2024-09-18T00:42:29.451 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.115934.log.gz 2024-09-18T00:42:29.452 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58344.log 2024-09-18T00:42:29.452 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139212.log 2024-09-18T00:42:29.453 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.96822.log.gz 2024-09-18T00:42:29.453 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.58344.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148904.log 2024-09-18T00:42:29.453 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139212.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65684.log 2024-09-18T00:42:29.454 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101846.log 2024-09-18T00:42:29.454 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65684.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83734.log 2024-09-18T00:42:29.455 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88584.log 2024-09-18T00:42:29.456 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83734.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111352.log.gz 2024-09-18T00:42:29.456 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110931.log 2024-09-18T00:42:29.456 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42744.log 2024-09-18T00:42:29.457 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92137.log 2024-09-18T00:42:29.457 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42744.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130759.log 2024-09-18T00:42:29.457 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48395.log.gz 2024-09-18T00:42:29.458 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92137.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152402.log 2024-09-18T00:42:29.458 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.130759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125025.log 2024-09-18T00:42:29.459 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152402.log: gzip 90.1% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.115218.log.gz 2024-09-18T00:42:29.459 INFO:teuthology.orchestra.run.smithi192.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.70986.log 2024-09-18T00:42:29.459 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138414.log 2024-09-18T00:42:29.460 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70986.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104022.log 2024-09-18T00:42:29.462 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138414.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149954.log 2024-09-18T00:42:29.462 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104022.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122913.log.gz 2024-09-18T00:42:29.462 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72811.log 2024-09-18T00:42:29.462 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43543.log 2024-09-18T00:42:29.463 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89211.log 2024-09-18T00:42:29.463 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43543.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141724.log 2024-09-18T00:42:29.463 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71756.log 2024-09-18T00:42:29.463 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69403.log.gz 2024-09-18T00:42:29.464 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129766.log 2024-09-18T00:42:29.464 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41544.log 2024-09-18T00:42:29.465 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129766.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34777.log 2024-09-18T00:42:29.465 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.70872.log.gz 2024-09-18T00:42:29.465 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79236.log.gz 2024-09-18T00:42:29.466 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41544.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81887.log 2024-09-18T00:42:29.466 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.34777.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34777.log.gz 2024-09-18T00:42:29.466 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86644.log 2024-09-18T00:42:29.467 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.81887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96453.log 2024-09-18T00:42:29.467 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79504.log 2024-09-18T00:42:29.468 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44331.log 2024-09-18T00:42:29.469 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79504.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66926.log 2024-09-18T00:42:29.469 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44331.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44331.log.gz 2024-09-18T00:42:29.469 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126292.log 2024-09-18T00:42:29.469 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111647.log 2024-09-18T00:42:29.470 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126292.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30664.log 2024-09-18T00:42:29.470 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108385.log 2024-09-18T00:42:29.471 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.30664.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30664.log.gz 2024-09-18T00:42:29.471 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96224.log 2024-09-18T00:42:29.471 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108385.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.115941.log.gz 2024-09-18T00:42:29.472 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109967.log 2024-09-18T00:42:29.474 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53349.log 2024-09-18T00:42:29.474 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95535.log 2024-09-18T00:42:29.474 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.127718.log.gz 2024-09-18T00:42:29.474 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53349.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131499.log 2024-09-18T00:42:29.475 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101383.log 2024-09-18T00:42:29.475 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131499.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57647.log 2024-09-18T00:42:29.476 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48941.log 2024-09-18T00:42:29.476 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99579.log 2024-09-18T00:42:29.477 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131827.log 2024-09-18T00:42:29.477 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99579.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53960.log.gz 2024-09-18T00:42:29.477 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69909.log 2024-09-18T00:42:29.478 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131827.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35539.log 2024-09-18T00:42:29.478 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69909.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.69909.log.gz 2024-09-18T00:42:29.478 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102100.log 2024-09-18T00:42:29.479 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.35539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52718.log 2024-09-18T00:42:29.480 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62471.log 2024-09-18T00:42:29.480 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136584.log 2024-09-18T00:42:29.481 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76150.log 2024-09-18T00:42:29.481 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119393.log 2024-09-18T00:42:29.482 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76150.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109963.log 2024-09-18T00:42:29.482 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39834.log.gz 2024-09-18T00:42:29.482 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.119393.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111561.log 2024-09-18T00:42:29.483 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145587.log 2024-09-18T00:42:29.484 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86997.log 2024-09-18T00:42:29.484 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39549.log.gz 2024-09-18T00:42:29.484 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145587.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106499.log 2024-09-18T00:42:29.484 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86997.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143588.log 2024-09-18T00:42:29.485 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.106499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84653.log 2024-09-18T00:42:29.485 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143588.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119827.log 2024-09-18T00:42:29.486 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123938.log 2024-09-18T00:42:29.486 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119827.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150103.log 2024-09-18T00:42:29.487 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145734.log 2024-09-18T00:42:29.487 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130654.log 2024-09-18T00:42:29.488 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145734.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84642.log 2024-09-18T00:42:29.488 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136481.log 2024-09-18T00:42:29.489 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114501.log 2024-09-18T00:42:29.489 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74120.log.gz 2024-09-18T00:42:29.489 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121378.log 2024-09-18T00:42:29.490 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114501.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71472.log 2024-09-18T00:42:29.490 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121378.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111851.log 2024-09-18T00:42:29.490 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71472.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145247.log 2024-09-18T00:42:29.491 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113951.log 2024-09-18T00:42:29.492 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151002.log 2024-09-18T00:42:29.492 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149264.log 2024-09-18T00:42:29.493 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100753.log 2024-09-18T00:42:29.493 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149264.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114372.log 2024-09-18T00:42:29.494 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116438.log 2024-09-18T00:42:29.494 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100416.log 2024-09-18T00:42:29.494 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116438.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37184.log 2024-09-18T00:42:29.495 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53287.log 2024-09-18T00:42:29.495 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37184.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77323.log 2024-09-18T00:42:29.496 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118606.log 2024-09-18T00:42:29.496 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43472.log 2024-09-18T00:42:29.497 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39941.log 2024-09-18T00:42:29.498 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43472.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.93249.log.gz 2024-09-18T00:42:29.498 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.93247.log.gz 2024-09-18T00:42:29.498 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114922.log 2024-09-18T00:42:29.498 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130559.log 2024-09-18T00:42:29.499 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57121.log 2024-09-18T00:42:29.500 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130559.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43165.log 2024-09-18T00:42:29.500 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.57121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59669.log 2024-09-18T00:42:29.500 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148002.log 2024-09-18T00:42:29.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137112.log 2024-09-18T00:42:29.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70758.log 2024-09-18T00:42:29.502 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97917.log 2024-09-18T00:42:29.502 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70758.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149114.log.gz 2024-09-18T00:42:29.502 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126934.log 2024-09-18T00:42:29.503 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97917.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125619.log 2024-09-18T00:42:29.503 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57654.log 2024-09-18T00:42:29.504 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125619.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101007.log 2024-09-18T00:42:29.504 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109753.log 2024-09-18T00:42:29.505 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101007.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153266.log 2024-09-18T00:42:29.505 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77472.log 2024-09-18T00:42:29.506 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.153266.log: 66.1% -- replaced with /var/log/ceph/cluster1-client.mirror.153266.log.gz 2024-09-18T00:42:29.506 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134865.log 2024-09-18T00:42:29.507 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77472.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42492.log 2024-09-18T00:42:29.507 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134865.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.35913.log.gz 2024-09-18T00:42:29.507 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57325.log 2024-09-18T00:42:29.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42492.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81662.log 2024-09-18T00:42:29.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57325.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129880.log 2024-09-18T00:42:29.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81662.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81662.log.gz 2024-09-18T00:42:29.509 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50829.log 2024-09-18T00:42:29.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71668.log 2024-09-18T00:42:29.510 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50829.log: 90.9% -- replaced with /var/log/ceph/cluster1-client.admin.70671.log.gz 2024-09-18T00:42:29.510 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119802.log 2024-09-18T00:42:29.511 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64254.log 2024-09-18T00:42:29.511 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75047.log 2024-09-18T00:42:29.511 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64254.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101047.log 2024-09-18T00:42:29.512 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75047.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88616.log 2024-09-18T00:42:29.513 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101047.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137576.log 2024-09-18T00:42:29.513 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43753.log 2024-09-18T00:42:29.513 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82478.log 2024-09-18T00:42:29.514 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138928.log 2024-09-18T00:42:29.514 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.82478.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129720.log 2024-09-18T00:42:29.515 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.138928.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136481.log.gz 2024-09-18T00:42:29.515 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93245.log 2024-09-18T00:42:29.515 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.136484.log.gz 2024-09-18T00:42:29.516 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52466.log 2024-09-18T00:42:29.516 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93245.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136584.log.gz 2024-09-18T00:42:29.517 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110092.log 2024-09-18T00:42:29.517 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52466.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89273.log 2024-09-18T00:42:29.517 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82001.log 2024-09-18T00:42:29.518 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58184.log 2024-09-18T00:42:29.518 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136691.log.gz 2024-09-18T00:42:29.518 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.82001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59491.log 2024-09-18T00:42:29.519 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.58184.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138891.log 2024-09-18T00:42:29.520 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.59491.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107985.log 2024-09-18T00:42:29.520 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55112.log 2024-09-18T00:42:29.521 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42022.log 2024-09-18T00:42:29.521 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.125368.log.gz 2024-09-18T00:42:29.522 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126888.log 2024-09-18T00:42:29.522 INFO:teuthology.orchestra.run.smithi192.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.admin.44702.log.gz 2024-09-18T00:42:29.522 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72671.log 2024-09-18T00:42:29.523 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126888.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67029.log 2024-09-18T00:42:29.524 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64591.log 2024-09-18T00:42:29.524 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53180.log 2024-09-18T00:42:29.525 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64591.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136900.log 2024-09-18T00:42:29.525 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37282.log.gz 2024-09-18T00:42:29.525 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53180.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67575.log 2024-09-18T00:42:29.526 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136900.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106653.log 2024-09-18T00:42:29.526 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44710.log 2024-09-18T00:42:29.527 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107462.log 2024-09-18T00:42:29.528 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126503.log 2024-09-18T00:42:29.528 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148843.log 2024-09-18T00:42:29.532 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101637.log 2024-09-18T00:42:29.532 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148843.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93903.log 2024-09-18T00:42:29.532 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148842.log 2024-09-18T00:42:29.532 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77472.log.gz 2024-09-18T00:42:29.532 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.93903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131569.log 2024-09-18T00:42:29.532 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129652.log 2024-09-18T00:42:29.533 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131569.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74203.log 2024-09-18T00:42:29.533 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122219.log 2024-09-18T00:42:29.533 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74203.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87911.log 2024-09-18T00:42:29.533 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86583.log.gz 2024-09-18T00:42:29.534 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87021.log 2024-09-18T00:42:29.534 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79154.log 2024-09-18T00:42:29.534 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96530.log 2024-09-18T00:42:29.534 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79154.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106634.log 2024-09-18T00:42:29.535 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96530.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47300.log 2024-09-18T00:42:29.535 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106634.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93743.log 2024-09-18T00:42:29.536 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47300.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73344.log 2024-09-18T00:42:29.536 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78415.log.gz 2024-09-18T00:42:29.536 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.93743.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93639.log 2024-09-18T00:42:29.537 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73344.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118907.log 2024-09-18T00:42:29.537 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93639.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.93639.log.gz 2024-09-18T00:42:29.537 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86375.log 2024-09-18T00:42:29.538 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75254.log 2024-09-18T00:42:29.538 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86375.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.151820.log 2024-09-18T00:42:29.539 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105437.log 2024-09-18T00:42:29.539 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109967.log.gz 2024-09-18T00:42:29.540 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109963.log.gz 2024-09-18T00:42:29.540 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.151820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138349.log 2024-09-18T00:42:29.540 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.105437.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105437.log.gz 2024-09-18T00:42:29.541 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62238.log 2024-09-18T00:42:29.541 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138349.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80716.log 2024-09-18T00:42:29.542 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62238.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110092.log.gz 2024-09-18T00:42:29.542 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42953.log 2024-09-18T00:42:29.542 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37323.log 2024-09-18T00:42:29.543 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35283.log 2024-09-18T00:42:29.544 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60260.log 2024-09-18T00:42:29.544 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.35283.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49270.log.gz 2024-09-18T00:42:29.544 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148693.log 2024-09-18T00:42:29.545 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72607.log 2024-09-18T00:42:29.545 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125386.log 2024-09-18T00:42:29.546 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72607.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88960.log 2024-09-18T00:42:29.546 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125386.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90011.log 2024-09-18T00:42:29.547 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108385.log.gz 2024-09-18T00:42:29.547 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88960.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148061.log 2024-09-18T00:42:29.547 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90011.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153152.log 2024-09-18T00:42:29.548 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72214.log 2024-09-18T00:42:29.549 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.153152.log: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153152.log.gz 2024-09-18T00:42:29.549 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38665.log 2024-09-18T00:42:29.549 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134361.log 2024-09-18T00:42:29.550 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38665.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120469.log 2024-09-18T00:42:29.550 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76065.log.gz 2024-09-18T00:42:29.550 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77575.log 2024-09-18T00:42:29.551 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120469.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43262.log 2024-09-18T00:42:29.551 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121736.log 2024-09-18T00:42:29.552 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42702.log 2024-09-18T00:42:29.552 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68864.log 2024-09-18T00:42:29.553 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133250.log 2024-09-18T00:42:29.553 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.68864.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101257.log.gz 2024-09-18T00:42:29.554 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55658.log 2024-09-18T00:42:29.554 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133250.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83777.log 2024-09-18T00:42:29.554 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55658.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114167.log 2024-09-18T00:42:29.555 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.83777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53413.log 2024-09-18T00:42:29.556 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116548.log 2024-09-18T00:42:29.556 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89881.log.gz 2024-09-18T00:42:29.556 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141496.log 2024-09-18T00:42:29.557 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.116548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147352.log 2024-09-18T00:42:29.558 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135138.log 2024-09-18T00:42:29.559 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93256.log 2024-09-18T00:42:29.559 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101088.log.gz 2024-09-18T00:42:29.559 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101090.log.gz 2024-09-18T00:42:29.559 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42742.log 2024-09-18T00:42:29.560 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62659.log 2024-09-18T00:42:29.560 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131757.log 2024-09-18T00:42:29.560 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59705.log 2024-09-18T00:42:29.561 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36774.log 2024-09-18T00:42:29.562 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116228.log 2024-09-18T00:42:29.562 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103294.log 2024-09-18T00:42:29.562 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135347.log 2024-09-18T00:42:29.563 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.103294.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69309.log 2024-09-18T00:42:29.563 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59953.log 2024-09-18T00:42:29.563 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.69309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148506.log 2024-09-18T00:42:29.564 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100880.log 2024-09-18T00:42:29.565 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124151.log.gz 2024-09-18T00:42:29.565 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51331.log 2024-09-18T00:42:29.565 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99702.log 2024-09-18T00:42:29.566 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47414.log 2024-09-18T00:42:29.566 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99702.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68954.log 2024-09-18T00:42:29.567 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47414.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50533.log 2024-09-18T00:42:29.567 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.68954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38919.log 2024-09-18T00:42:29.568 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119107.log 2024-09-18T00:42:29.569 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118383.log 2024-09-18T00:42:29.569 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.119107.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33601.log 2024-09-18T00:42:29.569 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42953.log.gz 2024-09-18T00:42:29.570 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.118383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100038.log 2024-09-18T00:42:29.570 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33601.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33601.log.gz 2024-09-18T00:42:29.570 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90512.log 2024-09-18T00:42:29.570 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100038.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148504.log 2024-09-18T00:42:29.571 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76631.log 2024-09-18T00:42:29.571 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148504.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43753.log.gz 2024-09-18T00:42:29.571 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140993.log 2024-09-18T00:42:29.572 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76631.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135685.log 2024-09-18T00:42:29.573 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.140993.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.117980.log.gz 2024-09-18T00:42:29.573 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146321.log 2024-09-18T00:42:29.573 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86376.log 2024-09-18T00:42:29.574 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146321.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45304.log 2024-09-18T00:42:29.574 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86376.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46455.log 2024-09-18T00:42:29.575 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76761.log 2024-09-18T00:42:29.575 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46455.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46455.log.gz 2024-09-18T00:42:29.576 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43472.log.gz 2024-09-18T00:42:29.576 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91022.log 2024-09-18T00:42:29.577 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76761.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.93245.log.gz 2024-09-18T00:42:29.577 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80721.log 2024-09-18T00:42:29.577 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43543.log.gz 2024-09-18T00:42:29.577 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91022.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128878.log 2024-09-18T00:42:29.578 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.80721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-18T00:42:29.578 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128878.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128878.log.gz 2024-09-18T00:42:29.578 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113071.log 2024-09-18T00:42:29.579 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-osd.0.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115616.log.gz 2024-09-18T00:42:29.579 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145483.log 2024-09-18T00:42:29.580 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113071.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77116.log 2024-09-18T00:42:29.580 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99159.log 2024-09-18T00:42:29.581 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77116.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153038.log 2024-09-18T00:42:29.581 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75464.log 2024-09-18T00:42:29.582 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.153038.log: 66.3% -- replaced with /var/log/ceph/cluster1-client.mirror.153038.log.gz 2024-09-18T00:42:29.582 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85698.log 2024-09-18T00:42:29.582 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75464.log: gzip 90.4% -5 --verbose -- replaced with /var/log/ceph/cluster1-client.mirror.41544.log.gz -- 2024-09-18T00:42:29.583 INFO:teuthology.orchestra.run.smithi192.stderr: /var/log/ceph/cluster1-client.mirror.134782.log 2024-09-18T00:42:29.583 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50407.log 2024-09-18T00:42:29.584 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39005.log 2024-09-18T00:42:29.585 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50407.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55112.log.gz 2024-09-18T00:42:29.585 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105541.log 2024-09-18T00:42:29.585 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39005.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134297.log 2024-09-18T00:42:29.586 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.105541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66987.log 2024-09-18T00:42:29.586 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134297.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51919.log 2024-09-18T00:42:29.587 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121378.log.gz 2024-09-18T00:42:29.587 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66987.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90390.log 2024-09-18T00:42:29.588 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51919.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.154319.log 2024-09-18T00:42:29.588 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90390.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.83777.log.gz 2024-09-18T00:42:29.589 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125661.log 2024-09-18T00:42:29.589 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.154319.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.154319.log.gz 2024-09-18T00:42:29.589 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147768.log 2024-09-18T00:42:29.589 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.154529.log 2024-09-18T00:42:29.590 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126587.log 2024-09-18T00:42:29.591 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.154529.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.154529.log.gz 2024-09-18T00:42:29.591 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139519.log 2024-09-18T00:42:29.591 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121120.log.gz 2024-09-18T00:42:29.591 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126587.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.147768.log.gz 2024-09-18T00:42:29.592 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39339.log 2024-09-18T00:42:29.592 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50869.log 2024-09-18T00:42:29.593 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116094.log 2024-09-18T00:42:29.593 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50869.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71472.log.gz 2024-09-18T00:42:29.593 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56209.log 2024-09-18T00:42:29.594 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111771.log 2024-09-18T00:42:29.594 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.56209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47575.log 2024-09-18T00:42:29.595 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94977.log 2024-09-18T00:42:29.595 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.47575.log: gzip -5 --verbose -- 89.9% /var/log/ceph/cluster1-client.mirror.123838.log 2024-09-18T00:42:29.596 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.72214.log.gz 2024-09-18T00:42:29.596 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.94977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100651.log 2024-09-18T00:42:29.597 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142993.log 2024-09-18T00:42:29.599 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138606.log 2024-09-18T00:42:29.599 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.142993.log: gzip -5 --verbose 87.6% -- /var/log/ceph/cluster1-client.mirror.40854.log 2024-09-18T00:42:29.599 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.140313.log.gz 2024-09-18T00:42:29.599 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138606.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.140315.log.gz 2024-09-18T00:42:29.599 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72449.log 2024-09-18T00:42:29.600 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40854.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133815.log 2024-09-18T00:42:29.600 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63877.log 2024-09-18T00:42:29.601 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39149.log 2024-09-18T00:42:29.601 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91684.log 2024-09-18T00:42:29.602 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39149.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106225.log 2024-09-18T00:42:29.602 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91684.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153655.log 2024-09-18T00:42:29.602 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.106225.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42072.log.gz 2024-09-18T00:42:29.603 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36185.log 2024-09-18T00:42:29.604 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.153655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36687.log 2024-09-18T00:42:29.604 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.admin.153655.log.gz 2024-09-18T00:42:29.604 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36112.log 2024-09-18T00:42:29.605 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133203.log 2024-09-18T00:42:29.605 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70074.log 2024-09-18T00:42:29.606 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133203.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144428.log 2024-09-18T00:42:29.606 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70074.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50827.log 2024-09-18T00:42:29.606 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106567.log 2024-09-18T00:42:29.607 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50827.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90767.log 2024-09-18T00:42:29.607 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135220.log.gz 2024-09-18T00:42:29.608 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.106567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46468.log 2024-09-18T00:42:29.608 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90767.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129424.log 2024-09-18T00:42:29.608 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46468.log: 89.9% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46468.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.135282.log.gz 2024-09-18T00:42:29.609 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:29.609 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69004.log 2024-09-18T00:42:29.610 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88899.log 2024-09-18T00:42:29.610 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69004.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47030.log 2024-09-18T00:42:29.611 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129150.log 2024-09-18T00:42:29.611 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67029.log.gz 2024-09-18T00:42:29.611 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.47030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145356.log 2024-09-18T00:42:29.612 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129150.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55239.log 2024-09-18T00:42:29.612 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104853.log 2024-09-18T00:42:29.613 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55239.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138349.log.gz 2024-09-18T00:42:29.613 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65956.log 2024-09-18T00:42:29.614 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70899.log 2024-09-18T00:42:29.614 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138414.log.gz 2024-09-18T00:42:29.615 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65956.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36993.log 2024-09-18T00:42:29.615 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97730.log 2024-09-18T00:42:29.616 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52741.log 2024-09-18T00:42:29.616 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97730.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141469.log 2024-09-18T00:42:29.617 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133368.log 2024-09-18T00:42:29.617 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141469.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34567.log 2024-09-18T00:42:29.618 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133368.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109230.log 2024-09-18T00:42:29.618 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.34567.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34567.log.gz 2024-09-18T00:42:29.619 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142888.log 2024-09-18T00:42:29.619 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109230.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151718.log 2024-09-18T00:42:29.620 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142888.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79700.log 2024-09-18T00:42:29.620 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.151718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130957.log 2024-09-18T00:42:29.621 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79700.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.79700.log.gz 2024-09-18T00:42:29.621 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66337.log 2024-09-18T00:42:29.621 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.44710.log.gz 2024-09-18T00:42:29.622 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67278.log 2024-09-18T00:42:29.623 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66337.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.45304.log.gz 2024-09-18T00:42:29.623 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81353.log 2024-09-18T00:42:29.623 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67278.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151334.log 2024-09-18T00:42:29.624 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51755.log 2024-09-18T00:42:29.624 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60449.log 2024-09-18T00:42:29.625 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51755.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126587.log.gz 2024-09-18T00:42:29.625 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116227.log 2024-09-18T00:42:29.625 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130453.log 2024-09-18T00:42:29.626 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58479.log 2024-09-18T00:42:29.626 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43542.log 2024-09-18T00:42:29.627 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.58479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142299.log 2024-09-18T00:42:29.627 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43542.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128578.log 2024-09-18T00:42:29.628 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134782.log.gz 2024-09-18T00:42:29.628 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112695.log 2024-09-18T00:42:29.629 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.128578.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96473.log 2024-09-18T00:42:29.630 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112695.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.154571.log 2024-09-18T00:42:29.630 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96473.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48081.log 2024-09-18T00:42:29.630 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40319.log.gz 2024-09-18T00:42:29.630 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.154571.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.154571.log.gz 2024-09-18T00:42:29.630 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144850.log 2024-09-18T00:42:29.631 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48081.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81319.log 2024-09-18T00:42:29.632 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63205.log 2024-09-18T00:42:29.633 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.81319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122637.log 2024-09-18T00:42:29.633 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63205.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56210.log.gz 2024-09-18T00:42:29.633 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87826.log 2024-09-18T00:42:29.633 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37998.log 2024-09-18T00:42:29.634 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62955.log 2024-09-18T00:42:29.635 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37998.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81610.log 2024-09-18T00:42:29.635 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46957.log 2024-09-18T00:42:29.636 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81610.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81610.log.gz 2024-09-18T00:42:29.636 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103404.log 2024-09-18T00:42:29.637 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.46957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73343.log 2024-09-18T00:42:29.637 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103404.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.103404.log.gz 2024-09-18T00:42:29.637 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75761.log 2024-09-18T00:42:29.638 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73343.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.86375.log.gz 2024-09-18T00:42:29.638 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112278.log 2024-09-18T00:42:29.638 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75761.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.140812.log 2024-09-18T00:42:29.639 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112278.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148318.log 2024-09-18T00:42:29.640 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.140812.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105101.log 2024-09-18T00:42:29.640 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56905.log 2024-09-18T00:42:29.641 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79046.log.gz 2024-09-18T00:42:29.641 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.105101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47808.log 2024-09-18T00:42:29.642 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.56905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89591.log 2024-09-18T00:42:29.642 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47808.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.152402.log.gz 2024-09-18T00:42:29.642 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86891.log 2024-09-18T00:42:29.643 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94681.log 2024-09-18T00:42:29.643 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119064.log 2024-09-18T00:42:29.643 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.94681.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150372.log 2024-09-18T00:42:29.644 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101092.log 2024-09-18T00:42:29.645 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150372.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117332.log 2024-09-18T00:42:29.645 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100880.log.gz 2024-09-18T00:42:29.645 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81345.log 2024-09-18T00:42:29.646 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.117332.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.117332.log.gz 2024-09-18T00:42:29.646 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.151334.log.gz 2024-09-18T00:42:29.646 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145911.log 2024-09-18T00:42:29.647 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81345.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98948.log.gz 2024-09-18T00:42:29.647 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.140993.log.gz 2024-09-18T00:42:29.647 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131737.log 2024-09-18T00:42:29.648 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138694.log 2024-09-18T00:42:29.648 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131737.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106157.log 2024-09-18T00:42:29.649 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.138694.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.138694.log.gz 2024-09-18T00:42:29.649 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123436.log 2024-09-18T00:42:29.649 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.106157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76105.log 2024-09-18T00:42:29.650 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123436.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117544.log 2024-09-18T00:42:29.650 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105587.log 2024-09-18T00:42:29.651 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.117544.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116922.log 2024-09-18T00:42:29.652 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.105587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113282.log 2024-09-18T00:42:29.652 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.116922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104635.log 2024-09-18T00:42:29.652 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113282.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40599.log 2024-09-18T00:42:29.653 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55366.log 2024-09-18T00:42:29.653 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40599.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97857.log 2024-09-18T00:42:29.654 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99198.log 2024-09-18T00:42:29.655 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91022.log.gz 2024-09-18T00:42:29.655 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136972.log 2024-09-18T00:42:29.655 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99198.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108492.log 2024-09-18T00:42:29.656 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68055.log 2024-09-18T00:42:29.656 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108492.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58026.log 2024-09-18T00:42:29.657 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.68055.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146639.log 2024-09-18T00:42:29.657 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.58026.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.58026.log.gz 2024-09-18T00:42:29.657 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153681.log 2024-09-18T00:42:29.658 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146639.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53558.log 2024-09-18T00:42:29.659 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.153681.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.153681.log.gz 2024-09-18T00:42:29.659 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52931.log 2024-09-18T00:42:29.659 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72540.log 2024-09-18T00:42:29.660 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.146639.log.gz 2024-09-18T00:42:29.660 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52931.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.144428.log.gz 2024-09-18T00:42:29.660 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51101.log 2024-09-18T00:42:29.661 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95924.log 2024-09-18T00:42:29.662 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126332.log 2024-09-18T00:42:29.662 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95924.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75573.log 2024-09-18T00:42:29.662 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110721.log 2024-09-18T00:42:29.663 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75573.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92786.log 2024-09-18T00:42:29.663 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110721.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.81345.log.gz 2024-09-18T00:42:29.664 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60876.log 2024-09-18T00:42:29.664 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76761.log.gz 2024-09-18T00:42:29.664 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.92786.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97897.log 2024-09-18T00:42:29.665 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60876.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34107.log 2024-09-18T00:42:29.665 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97897.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126792.log 2024-09-18T00:42:29.666 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.34107.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34107.log.gz 2024-09-18T00:42:29.666 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50617.log 2024-09-18T00:42:29.666 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86954.log 2024-09-18T00:42:29.667 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117840.log 2024-09-18T00:42:29.668 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107500.log 2024-09-18T00:42:29.668 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109230.log.gz 2024-09-18T00:42:29.668 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.117840.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90390.log.gz 2024-09-18T00:42:29.668 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113702.log 2024-09-18T00:42:29.669 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107500.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54297.log 2024-09-18T00:42:29.669 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134797.log 2024-09-18T00:42:29.670 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54297.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39212.log 2024-09-18T00:42:29.671 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134797.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74834.log 2024-09-18T00:42:29.671 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39212.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104030.log 2024-09-18T00:42:29.672 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122763.log 2024-09-18T00:42:29.672 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.104030.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134633.log.gz 2024-09-18T00:42:29.672 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82716.log 2024-09-18T00:42:29.673 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115031.log 2024-09-18T00:42:29.673 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.82716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56063.log 2024-09-18T00:42:29.674 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.115031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100332.log 2024-09-18T00:42:29.674 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.56063.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86454.log 2024-09-18T00:42:29.675 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100122.log 2024-09-18T00:42:29.676 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86454.log: gzip 89.9% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.134297.log.gz --verbose 2024-09-18T00:42:29.676 INFO:teuthology.orchestra.run.smithi192.stderr: -- /var/log/ceph/cluster1-client.mirror.108703.log 2024-09-18T00:42:29.676 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133164.log 2024-09-18T00:42:29.677 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77032.log 2024-09-18T00:42:29.677 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133164.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153742.log 2024-09-18T00:42:29.678 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94380.log 2024-09-18T00:42:29.679 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.153742.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104640.log 2024-09-18T00:42:29.679 INFO:teuthology.orchestra.run.smithi192.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.153742.log.gz 2024-09-18T00:42:29.679 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.94380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44171.log 2024-09-18T00:42:29.680 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104640.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.119064.log.gz 2024-09-18T00:42:29.680 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110381.log 2024-09-18T00:42:29.680 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44171.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142515.log 2024-09-18T00:42:29.681 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110381.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121838.log 2024-09-18T00:42:29.681 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97730.log.gz 2024-09-18T00:42:29.682 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.142515.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150078.log 2024-09-18T00:42:29.682 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68477.log 2024-09-18T00:42:29.683 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150078.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.81194.log.gz 2024-09-18T00:42:29.683 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123836.log 2024-09-18T00:42:29.684 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68477.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.81319.log.gz 2024-09-18T00:42:29.684 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128007.log 2024-09-18T00:42:29.684 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123836.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56940.log 2024-09-18T00:42:29.684 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138606.log.gz 2024-09-18T00:42:29.685 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.128007.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78314.log 2024-09-18T00:42:29.686 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.56940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133876.log 2024-09-18T00:42:29.686 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112695.log.gz 2024-09-18T00:42:29.686 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98151.log 2024-09-18T00:42:29.687 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133876.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57595.log 2024-09-18T00:42:29.687 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98151.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57650.log 2024-09-18T00:42:29.688 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.57595.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141924.log 2024-09-18T00:42:29.688 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1%/var/log/ceph/cluster2-client.admin.57650.log: -- replaced with /var/log/ceph/cluster1-client.mirror.53348.log.gz 2024-09-18T00:42:29.689 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76762.log 2024-09-18T00:42:29.689 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.141924.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130869.log.gz 2024-09-18T00:42:29.689 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53476.log 2024-09-18T00:42:29.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76762.log: gzip -5 89.4% --verbose -- -- replaced with /var/log/ceph/cluster1-client.mirror.47808.log.gz /var/log/ceph/cluster1-client.mirror.39000.log 2024-09-18T00:42:29.690 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:29.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110178.log 2024-09-18T00:42:29.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39000.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53558.log.gz 2024-09-18T00:42:29.691 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140266.log 2024-09-18T00:42:29.692 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95313.log 2024-09-18T00:42:29.693 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.140266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125324.log 2024-09-18T00:42:29.693 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86754.log 2024-09-18T00:42:29.693 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121840.log 2024-09-18T00:42:29.694 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39859.log 2024-09-18T00:42:29.694 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.110721.log.gz 2024-09-18T00:42:29.694 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123476.log 2024-09-18T00:42:29.695 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39859.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119674.log 2024-09-18T00:42:29.696 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120952.log 2024-09-18T00:42:29.697 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50575.log 2024-09-18T00:42:29.697 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57454.log 2024-09-18T00:42:29.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50575.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69205.log 2024-09-18T00:42:29.698 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.150078.log.gz 2024-09-18T00:42:29.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125860.log 2024-09-18T00:42:29.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.69205.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151331.log 2024-09-18T00:42:29.699 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125860.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65200.log 2024-09-18T00:42:29.699 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106764.log 2024-09-18T00:42:29.700 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51521.log 2024-09-18T00:42:29.700 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106764.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.131758.log.gz 2024-09-18T00:42:29.700 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123015.log 2024-09-18T00:42:29.701 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49584.log 2024-09-18T00:42:29.702 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123015.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.130957.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59866.log 2024-09-18T00:42:29.702 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:29.702 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82042.log 2024-09-18T00:42:29.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103763.log 2024-09-18T00:42:29.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.82042.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137238.log 2024-09-18T00:42:29.704 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35590.log 2024-09-18T00:42:29.704 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137238.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72575.log 2024-09-18T00:42:29.706 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.35590.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146847.log 2024-09-18T00:42:29.706 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72575.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.106567.log.gz 2024-09-18T00:42:29.706 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92004.log 2024-09-18T00:42:29.706 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146847.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102937.log 2024-09-18T00:42:29.707 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79493.log 2024-09-18T00:42:29.708 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102937.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88457.log 2024-09-18T00:42:29.708 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79493.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.79493.log.gz 2024-09-18T00:42:29.709 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92096.log 2024-09-18T00:42:29.709 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133876.log.gz 2024-09-18T00:42:29.709 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88457.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103294.log.gz 2024-09-18T00:42:29.709 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52508.log 2024-09-18T00:42:29.710 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114122.log 2024-09-18T00:42:29.710 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52508.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107853.log 2024-09-18T00:42:29.710 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103492.log.gz 2024-09-18T00:42:29.711 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60107.log 2024-09-18T00:42:29.712 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142360.log 2024-09-18T00:42:29.712 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60107.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128506.log 2024-09-18T00:42:29.712 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68226.log 2024-09-18T00:42:29.713 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68353.log 2024-09-18T00:42:29.714 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126877.log 2024-09-18T00:42:29.714 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152808.log 2024-09-18T00:42:29.715 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.126877.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.126877.log.gz 2024-09-18T00:42:29.715 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90641.log 2024-09-18T00:42:29.715 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.152808.log: 66.3% -- replaced with /var/log/ceph/cluster1-client.mirror.152808.log.gz 2024-09-18T00:42:29.715 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43798.log 2024-09-18T00:42:29.716 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90641.log: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.69987.log.gz 2024-09-18T00:42:29.716 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111981.log 2024-09-18T00:42:29.717 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63540.log 2024-09-18T00:42:29.717 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111981.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101591.log 2024-09-18T00:42:29.718 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113322.log 2024-09-18T00:42:29.718 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48673.log.gz 2024-09-18T00:42:29.718 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75573.log.gz 2024-09-18T00:42:29.719 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101591.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101382.log 2024-09-18T00:42:29.720 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85738.log 2024-09-18T00:42:29.720 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76762.log.gz 2024-09-18T00:42:29.720 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68332.log 2024-09-18T00:42:29.720 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95013.log 2024-09-18T00:42:29.721 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68332.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91386.log.gz 2024-09-18T00:42:29.721 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83234.log 2024-09-18T00:42:29.721 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95013.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.91684.log.gz 2024-09-18T00:42:29.722 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125179.log 2024-09-18T00:42:29.722 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83234.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146695.log 2024-09-18T00:42:29.723 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112191.log 2024-09-18T00:42:29.723 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146695.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138080.log 2024-09-18T00:42:29.724 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96510.log 2024-09-18T00:42:29.724 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138080.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.146695.log.gz 2024-09-18T00:42:29.725 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133034.log 2024-09-18T00:42:29.725 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123728.log 2024-09-18T00:42:29.726 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107615.log 2024-09-18T00:42:29.727 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52635.log 2024-09-18T00:42:29.727 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83022.log 2024-09-18T00:42:29.728 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52635.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128695.log 2024-09-18T00:42:29.728 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123939.log 2024-09-18T00:42:29.729 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128695.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72832.log 2024-09-18T00:42:29.730 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123939.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120816.log 2024-09-18T00:42:29.730 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.86954.log.gz 2024-09-18T00:42:29.730 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40011.log 2024-09-18T00:42:29.731 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120816.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52509.log 2024-09-18T00:42:29.731 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40011.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140734.log 2024-09-18T00:42:29.732 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35249.log 2024-09-18T00:42:29.733 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140734.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140734.log.gz 2024-09-18T00:42:29.733 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.145356.log.gz 2024-09-18T00:42:29.733 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140760.log 2024-09-18T00:42:29.733 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.35249.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134004.log 2024-09-18T00:42:29.734 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140760.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140760.log.gz 2024-09-18T00:42:29.734 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43912.log 2024-09-18T00:42:29.735 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91488.log 2024-09-18T00:42:29.735 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43912.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58572.log 2024-09-18T00:42:29.736 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91488.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139722.log 2024-09-18T00:42:29.736 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.58572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73282.log 2024-09-18T00:42:29.737 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47720.log 2024-09-18T00:42:29.737 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73282.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145458.log 2024-09-18T00:42:29.738 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.47720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59601.log 2024-09-18T00:42:29.738 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145458.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69829.log 2024-09-18T00:42:29.739 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86352.log 2024-09-18T00:42:29.739 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.99326.log.gz 2024-09-18T00:42:29.740 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69829.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.69829.log.gz 2024-09-18T00:42:29.740 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36469.log 2024-09-18T00:42:29.741 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.35653.log 2024-09-18T00:42:29.741 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36469.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36088.log 2024-09-18T00:42:29.741 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.0.35653.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104013.log 2024-09-18T00:42:29.741 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80366.log 2024-09-18T00:42:29.742 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.151331.log.gz 2024-09-18T00:42:29.742 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62366.log 2024-09-18T00:42:29.743 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88457.log.gz 2024-09-18T00:42:29.743 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.80366.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.25640.log 2024-09-18T00:42:29.744 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144505.log 2024-09-18T00:42:29.744 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ceph.tmp-client.admin.25640.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.25640.log.gz 2024-09-18T00:42:29.744 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117451.log 2024-09-18T00:42:29.745 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94722.log 2024-09-18T00:42:29.746 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.117451.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63459.log 2024-09-18T00:42:29.746 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.94722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102780.log 2024-09-18T00:42:29.747 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63459.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145776.log 2024-09-18T00:42:29.747 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102780.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119648.log 2024-09-18T00:42:29.748 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83928.log 2024-09-18T00:42:29.749 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101299.log 2024-09-18T00:42:29.749 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108760.log.gz 2024-09-18T00:42:29.749 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.83928.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48520.log 2024-09-18T00:42:29.750 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101299.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69414.log 2024-09-18T00:42:29.750 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41232.log 2024-09-18T00:42:29.751 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.69414.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42703.log 2024-09-18T00:42:29.751 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78314.log.gz 2024-09-18T00:42:29.751 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146533.log 2024-09-18T00:42:29.752 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137720.log 2024-09-18T00:42:29.752 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94872.log 2024-09-18T00:42:29.753 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35437.log 2024-09-18T00:42:29.753 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.94872.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70583.log 2024-09-18T00:42:29.754 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.35437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148503.log 2024-09-18T00:42:29.755 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110381.log.gz/var/log/ceph/cluster2-client.mirror.70583.log: 2024-09-18T00:42:29.755 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60913.log 2024-09-18T00:42:29.755 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75550.log 2024-09-18T00:42:29.756 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60913.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96847.log 2024-09-18T00:42:29.757 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121505.log 2024-09-18T00:42:29.757 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96847.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71604.log 2024-09-18T00:42:29.758 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62576.log.gz 2024-09-18T00:42:29.758 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53809.log 2024-09-18T00:42:29.758 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71604.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130088.log 2024-09-18T00:42:29.759 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53809.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62658.log.gz 2024-09-18T00:42:29.759 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44004.log 2024-09-18T00:42:29.760 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.130088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96365.log 2024-09-18T00:42:29.760 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47004.log 2024-09-18T00:42:29.761 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67407.log 2024-09-18T00:42:29.761 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.47004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107312.log 2024-09-18T00:42:29.761 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.117451.log.gz 2024-09-18T00:42:29.762 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56340.log 2024-09-18T00:42:29.763 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44709.log 2024-09-18T00:42:29.763 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152767.log 2024-09-18T00:42:29.764 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44709.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152210.log 2024-09-18T00:42:29.764 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.152767.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52140.log 2024-09-18T00:42:29.765 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.152767.log.gz 2024-09-18T00:42:29.765 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55616.log 2024-09-18T00:42:29.765 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51037.log 2024-09-18T00:42:29.766 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137321.log 2024-09-18T00:42:29.766 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51037.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52508.log.gz 2024-09-18T00:42:29.767 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36045.log 2024-09-18T00:42:29.767 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52509.log.gz 2024-09-18T00:42:29.767 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87870.log 2024-09-18T00:42:29.768 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51668.log 2024-09-18T00:42:29.769 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113661.log 2024-09-18T00:42:29.769 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118737.log 2024-09-18T00:42:29.770 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114372.log.gz 2024-09-18T00:42:29.770 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110804.log 2024-09-18T00:42:29.770 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71246.log 2024-09-18T00:42:29.771 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99788.log 2024-09-18T00:42:29.772 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71246.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81290.log 2024-09-18T00:42:29.772 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99788.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136610.log 2024-09-18T00:42:29.773 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.81290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47661.log 2024-09-18T00:42:29.773 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63708.log 2024-09-18T00:42:29.774 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133163.log 2024-09-18T00:42:29.774 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63708.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65599.log.gz 2024-09-18T00:42:29.775 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107158.log 2024-09-18T00:42:29.775 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42913.log 2024-09-18T00:42:29.775 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45445.log 2024-09-18T00:42:29.776 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42913.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147164.log 2024-09-18T00:42:29.777 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45445.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117927.log 2024-09-18T00:42:29.777 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58458.log 2024-09-18T00:42:29.778 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.117927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127306.log 2024-09-18T00:42:29.778 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.58458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95145.log 2024-09-18T00:42:29.779 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.147164.log.gz 2024-09-18T00:42:29.779 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127306.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75673.log 2024-09-18T00:42:29.780 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95145.log: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46957.log.gz 2024-09-18T00:42:29.780 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145777.log 2024-09-18T00:42:29.780 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51078.log 2024-09-18T00:42:29.781 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145777.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128137.log 2024-09-18T00:42:29.781 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51078.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63120.log 2024-09-18T00:42:29.782 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70168.log 2024-09-18T00:42:29.783 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63120.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128103.log 2024-09-18T00:42:29.783 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.70168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63583.log 2024-09-18T00:42:29.784 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.128103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50659.log 2024-09-18T00:42:29.784 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100332.log.gz 2024-09-18T00:42:29.785 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91168.log 2024-09-18T00:42:29.785 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.83022.log.gz 2024-09-18T00:42:29.785 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.35617.log 2024-09-18T00:42:29.786 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144851.log 2024-09-18T00:42:29.786 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.0.35617.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116997.log 2024-09-18T00:42:29.787 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91061.log 2024-09-18T00:42:29.787 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.116997.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58821.log 2024-09-18T00:42:29.788 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110342.log 2024-09-18T00:42:29.788 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.58821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50226.log 2024-09-18T00:42:29.789 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110342.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102853.log 2024-09-18T00:42:29.789 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53476.log.gz 2024-09-18T00:42:29.790 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40760.log 2024-09-18T00:42:29.790 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48698.log 2024-09-18T00:42:29.791 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40760.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140323.log 2024-09-18T00:42:29.791 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36728.log 2024-09-18T00:42:29.792 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111561.log.gz 2024-09-18T00:42:29.792 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138653.log 2024-09-18T00:42:29.793 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.36728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41139.log 2024-09-18T00:42:29.793 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101300.log 2024-09-18T00:42:29.794 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150637.log 2024-09-18T00:42:29.795 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101300.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86374.log 2024-09-18T00:42:29.795 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118363.log 2024-09-18T00:42:29.796 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50406.log 2024-09-18T00:42:29.796 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118363.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131091.log 2024-09-18T00:42:29.797 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50406.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74500.log 2024-09-18T00:42:29.797 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133480.log 2024-09-18T00:42:29.798 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74500.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.114909.log 2024-09-18T00:42:29.798 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133480.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98151.log.gz 2024-09-18T00:42:29.798 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.114977.log 2024-09-18T00:42:29.799 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.114909.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.114909.log.gz 2024-09-18T00:42:29.799 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57137.log 2024-09-18T00:42:29.799 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.114977.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.114977.log.gz 2024-09-18T00:42:29.800 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119496.log 2024-09-18T00:42:29.800 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71604.log.gz 2024-09-18T00:42:29.800 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113955.log 2024-09-18T00:42:29.802 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123307.log 2024-09-18T00:42:29.802 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71756.log.gz 2024-09-18T00:42:29.802 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113955.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71668.log.gz 2024-09-18T00:42:29.802 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85169.log 2024-09-18T00:42:29.802 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58894.log.gz 2024-09-18T00:42:29.803 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90488.log 2024-09-18T00:42:29.803 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69935.log 2024-09-18T00:42:29.804 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75130.log 2024-09-18T00:42:29.804 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69935.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.69935.log.gz 2024-09-18T00:42:29.804 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131453.log 2024-09-18T00:42:29.805 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53096.log 2024-09-18T00:42:29.805 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131453.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104988.log 2024-09-18T00:42:29.806 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53096.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66522.log.gz 2024-09-18T00:42:29.806 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128697.log 2024-09-18T00:42:29.807 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125493.log 2024-09-18T00:42:29.807 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128697.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152611.log 2024-09-18T00:42:29.808 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125493.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115543.log 2024-09-18T00:42:29.808 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152611.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.152611.log.gz 2024-09-18T00:42:29.809 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64549.log 2024-09-18T00:42:29.809 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.115543.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103024.log 2024-09-18T00:42:29.810 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64549.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122513.log 2024-09-18T00:42:29.811 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.103024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.152942.log 2024-09-18T00:42:29.811 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127131.log 2024-09-18T00:42:29.811 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.152942.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.152942.log.gz 2024-09-18T00:42:29.811 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80260.log 2024-09-18T00:42:29.812 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76843.log 2024-09-18T00:42:29.813 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.80260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67194.log 2024-09-18T00:42:29.813 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76843.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115259.log 2024-09-18T00:42:29.814 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44464.log 2024-09-18T00:42:29.814 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52197.log 2024-09-18T00:42:29.815 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44464.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44464.log.gz 2024-09-18T00:42:29.815 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111351.log 2024-09-18T00:42:29.815 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52197.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54231.log 2024-09-18T00:42:29.816 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111351.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64380.log.gz 2024-09-18T00:42:29.816 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80735.log 2024-09-18T00:42:29.816 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54231.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116093.log 2024-09-18T00:42:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.80735.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64424.log.gz 2024-09-18T00:42:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104638.log 2024-09-18T00:42:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125095.log 2024-09-18T00:42:29.818 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104638.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105997.log 2024-09-18T00:42:29.819 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125095.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64464.log.gzgzip 2024-09-18T00:42:29.819 INFO:teuthology.orchestra.run.smithi192.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.69071.log 2024-09-18T00:42:29.819 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64465.log.gz 2024-09-18T00:42:29.819 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.105997.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64507.log.gz 2024-09-18T00:42:29.820 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35505.log 2024-09-18T00:42:29.820 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39209.log.gz 2024-09-18T00:42:29.820 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69071.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65723.log 2024-09-18T00:42:29.820 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.35505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87460.log 2024-09-18T00:42:29.821 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64634.log 2024-09-18T00:42:29.822 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87460.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58914.log 2024-09-18T00:42:29.822 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64634.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130259.log 2024-09-18T00:42:29.823 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.58914.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54072.log 2024-09-18T00:42:29.823 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147138.log 2024-09-18T00:42:29.824 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55198.log 2024-09-18T00:42:29.824 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123477.log.gz 2024-09-18T00:42:29.825 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85637.log 2024-09-18T00:42:29.825 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55198.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136215.log 2024-09-18T00:42:29.826 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42954.log 2024-09-18T00:42:29.826 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113322.log.gz 2024-09-18T00:42:29.826 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52427.log 2024-09-18T00:42:29.827 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47903.log 2024-09-18T00:42:29.827 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.139029.log.gz 2024-09-18T00:42:29.828 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52427.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145879.log 2024-09-18T00:42:29.828 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121334.log 2024-09-18T00:42:29.829 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145879.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99703.log 2024-09-18T00:42:29.830 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66606.log 2024-09-18T00:42:29.830 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114376.log 2024-09-18T00:42:29.830 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87699.log 2024-09-18T00:42:29.831 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146928.log 2024-09-18T00:42:29.832 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87699.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105608.log 2024-09-18T00:42:29.832 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146928.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81351.log 2024-09-18T00:42:29.832 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.105608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86585.log 2024-09-18T00:42:29.833 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43912.log.gz 2024-09-18T00:42:29.833 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142383.log 2024-09-18T00:42:29.834 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86585.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40812.log 2024-09-18T00:42:29.834 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.142383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70624.log 2024-09-18T00:42:29.835 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40812.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119674.log.gz 2024-09-18T00:42:29.835 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143952.log 2024-09-18T00:42:29.836 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.70624.log: 66.4% -- replaced with /var/log/ceph/cluster1-client.admin.35249.log.gz 2024-09-18T00:42:29.836 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82620.log 2024-09-18T00:42:29.836 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117000.log 2024-09-18T00:42:29.837 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.82620.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63143.log 2024-09-18T00:42:29.837 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42426.log.gz/var/log/ceph/cluster2-client.admin.117000.log: 2024-09-18T00:42:29.838 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145902.log 2024-09-18T00:42:29.838 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63143.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57174.log 2024-09-18T00:42:29.839 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44005.log 2024-09-18T00:42:29.839 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.57174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137320.log 2024-09-18T00:42:29.840 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44005.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133312.log 2024-09-18T00:42:29.840 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95794.log 2024-09-18T00:42:29.841 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130736.log 2024-09-18T00:42:29.842 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.95794.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51525.log 2024-09-18T00:42:29.842 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130736.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84478.log 2024-09-18T00:42:29.843 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51525.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92482.log 2024-09-18T00:42:29.843 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69003.log 2024-09-18T00:42:29.844 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.92482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111222.log 2024-09-18T00:42:29.844 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121420.log 2024-09-18T00:42:29.845 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111222.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-18T00:42:29.845 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121420.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148272.log 2024-09-18T00:42:29.846 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67941.log 2024-09-18T00:42:29.846 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148272.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140721.log 2024-09-18T00:42:29.847 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103772.log 2024-09-18T00:42:29.848 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140721.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.148272.log.gz 2024-09-18T00:42:29.848 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140721.log.gz 2024-09-18T00:42:29.848 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126040.log 2024-09-18T00:42:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136270.log 2024-09-18T00:42:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126040.log: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.142735.log.gz 2024-09-18T00:42:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135114.log 2024-09-18T00:42:29.850 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99747.log 2024-09-18T00:42:29.850 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135114.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.142993.log.gz 2024-09-18T00:42:29.851 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68995.log 2024-09-18T00:42:29.851 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99747.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.143359.log.gz 2024-09-18T00:42:29.851 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123016.log 2024-09-18T00:42:29.852 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.68995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117907.log 2024-09-18T00:42:29.853 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123016.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.154445.log 2024-09-18T00:42:29.853 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.117907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71536.log 2024-09-18T00:42:29.854 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.154445.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.154445.log.gz 2024-09-18T00:42:29.854 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126786.log 2024-09-18T00:42:29.854 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71536.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136610.log.gz 2024-09-18T00:42:29.854 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90557.log 2024-09-18T00:42:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.126786.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.126786.log.gz 2024-09-18T00:42:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102352.log 2024-09-18T00:42:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41792.log 2024-09-18T00:42:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141382.log 2024-09-18T00:42:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41792.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80500.log 2024-09-18T00:42:29.858 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105970.log 2024-09-18T00:42:29.858 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80500.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57125.log 2024-09-18T00:42:29.859 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.105970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58365.log 2024-09-18T00:42:29.859 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.57125.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33643.log 2024-09-18T00:42:29.860 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.58365.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117772.log 2024-09-18T00:42:29.860 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.74834.log.gz 2024-09-18T00:42:29.860 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33643.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33643.log.gz 2024-09-18T00:42:29.860 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133812.log 2024-09-18T00:42:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.117772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91353.log 2024-09-18T00:42:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133812.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85738.log.gz 2024-09-18T00:42:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102353.log 2024-09-18T00:42:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43373.log 2024-09-18T00:42:29.863 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138434.log 2024-09-18T00:42:29.864 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147687.log 2024-09-18T00:42:29.864 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138434.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133312.log.gz 2024-09-18T00:42:29.865 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146592.log 2024-09-18T00:42:29.865 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133605.log 2024-09-18T00:42:29.865 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127313.log 2024-09-18T00:42:29.866 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133605.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126250.log 2024-09-18T00:42:29.867 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146592.log.gz 2024-09-18T00:42:29.867 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127313.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.145458.log.gz 2024-09-18T00:42:29.867 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136544.log 2024-09-18T00:42:29.868 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126250.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87763.log 2024-09-18T00:42:29.868 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136544.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104891.log 2024-09-18T00:42:29.869 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119964.log 2024-09-18T00:42:29.870 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87763.log: /var/log/ceph/cluster2-client.admin.104891.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48520.log.gz 2024-09-18T00:42:29.870 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93691.log 2024-09-18T00:42:29.871 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119964.log: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40760.log.gz 2024-09-18T00:42:29.871 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144725.log 2024-09-18T00:42:29.871 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93691.log: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55198.log.gz 2024-09-18T00:42:29.871 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126040.log.gz 2024-09-18T00:42:29.871 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.93691.log.gz 2024-09-18T00:42:29.872 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58157.log 2024-09-18T00:42:29.873 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144725.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125703.log 2024-09-18T00:42:29.873 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40599.log.gz 2024-09-18T00:42:29.873 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.58157.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34861.log 2024-09-18T00:42:29.875 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66336.log 2024-09-18T00:42:29.875 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.34861.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34861.log.gz 2024-09-18T00:42:29.875 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141905.log 2024-09-18T00:42:29.875 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39215.log.gz/var/log/ceph/cluster1-client.mirror.66336.log: 2024-09-18T00:42:29.875 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39339.log.gz 2024-09-18T00:42:29.876 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112612.log 2024-09-18T00:42:29.876 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.141905.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109975.log.gz 2024-09-18T00:42:29.877 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68674.log 2024-09-18T00:42:29.877 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107026.log 2024-09-18T00:42:29.877 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68674.log: gzip -5 --verbose -- 90.0% /var/log/ceph/cluster1-client.admin.47461.log 2024-09-18T00:42:29.877 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.82960.log.gz 2024-09-18T00:42:29.878 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103853.log 2024-09-18T00:42:29.878 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.47461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101760.log 2024-09-18T00:42:29.879 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103853.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.57647.log.gz 2024-09-18T00:42:29.879 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119460.log 2024-09-18T00:42:29.880 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130995.log 2024-09-18T00:42:29.880 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47118.log 2024-09-18T00:42:29.881 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98445.log 2024-09-18T00:42:29.881 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.47118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87620.log 2024-09-18T00:42:29.882 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98445.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89359.log 2024-09-18T00:42:29.882 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87620.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76464.log 2024-09-18T00:42:29.883 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89359.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52091.log 2024-09-18T00:42:29.884 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76464.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143952.log.gz 2024-09-18T00:42:29.884 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72063.log 2024-09-18T00:42:29.885 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107373.log 2024-09-18T00:42:29.885 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72063.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58800.log 2024-09-18T00:42:29.886 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111221.log 2024-09-18T00:42:29.886 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113955.log.gz/var/log/ceph/cluster2-client.mirror.58800.log: 2024-09-18T00:42:29.886 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67407.log.gz 2024-09-18T00:42:29.886 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44172.log 2024-09-18T00:42:29.887 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87954.log 2024-09-18T00:42:29.888 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45079.log 2024-09-18T00:42:29.889 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146007.log 2024-09-18T00:42:29.889 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.45079.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130259.log.gz 2024-09-18T00:42:29.889 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36812.log 2024-09-18T00:42:29.889 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146007.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92928.log 2024-09-18T00:42:29.890 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36812.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99747.log.gz 2024-09-18T00:42:29.890 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79278.log 2024-09-18T00:42:29.891 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92928.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137323.log 2024-09-18T00:42:29.892 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79278.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.58157.log.gz 2024-09-18T00:42:29.892 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70375.log 2024-09-18T00:42:29.892 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137323.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128971.log 2024-09-18T00:42:29.892 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112694.log 2024-09-18T00:42:29.893 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128971.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128971.log.gz 2024-09-18T00:42:29.893 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98987.log 2024-09-18T00:42:29.894 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112694.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50406.log.gz 2024-09-18T00:42:29.894 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80318.log 2024-09-18T00:42:29.895 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98987.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142071.log 2024-09-18T00:42:29.895 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.80318.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134720.log 2024-09-18T00:42:29.896 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142071.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92533.log 2024-09-18T00:42:29.897 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137030.log 2024-09-18T00:42:29.897 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92533.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87870.log.gz 2024-09-18T00:42:29.897 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138642.log 2024-09-18T00:42:29.898 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106980.log 2024-09-18T00:42:29.898 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138642.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92537.log 2024-09-18T00:42:29.899 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.106980.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152637.log 2024-09-18T00:42:29.900 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133122.log 2024-09-18T00:42:29.900 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152637.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.152637.log.gz 2024-09-18T00:42:29.900 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89719.log 2024-09-18T00:42:29.901 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133122.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134004.log.gz 2024-09-18T00:42:29.901 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121463.log 2024-09-18T00:42:29.901 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78416.log 2024-09-18T00:42:29.902 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121463.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149846.log 2024-09-18T00:42:29.903 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78416.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.131453.log.gz 2024-09-18T00:42:29.903 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65640.log 2024-09-18T00:42:29.904 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75253.log 2024-09-18T00:42:29.904 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98906.log 2024-09-18T00:42:29.905 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149450.log 2024-09-18T00:42:29.905 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80104.log 2024-09-18T00:42:29.906 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149450.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130843.log 2024-09-18T00:42:29.907 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.80104.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125450.log 2024-09-18T00:42:29.907 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130843.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.125095.log.gz 2024-09-18T00:42:29.907 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144938.log 2024-09-18T00:42:29.907 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125450.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109711.log 2024-09-18T00:42:29.908 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92195.log 2024-09-18T00:42:29.909 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109711.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70054.log 2024-09-18T00:42:29.910 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117308.log 2024-09-18T00:42:29.910 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.70054.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102937.log.gz 2024-09-18T00:42:29.910 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68011.log 2024-09-18T00:42:29.911 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.117308.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.117308.log.gz 2024-09-18T00:42:29.911 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122409.log 2024-09-18T00:42:29.911 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68011.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.68011.log.gz 2024-09-18T00:42:29.911 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80590.log 2024-09-18T00:42:29.912 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78835.log 2024-09-18T00:42:29.913 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80590.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105929.log 2024-09-18T00:42:29.913 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39731.log 2024-09-18T00:42:29.914 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.105929.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77410.log 2024-09-18T00:42:29.914 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39731.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42587.log 2024-09-18T00:42:29.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98677.log 2024-09-18T00:42:29.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42587.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45324.log 2024-09-18T00:42:29.916 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98677.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71474.log 2024-09-18T00:42:29.916 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115126.log 2024-09-18T00:42:29.917 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71474.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41862.log 2024-09-18T00:42:29.917 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115126.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.115126.log.gz 2024-09-18T00:42:29.917 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128444.log 2024-09-18T00:42:29.918 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41862.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65723.log.gz 2024-09-18T00:42:29.918 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.65684.log.gz 2024-09-18T00:42:29.918 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62869.log 2024-09-18T00:42:29.919 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128444.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69883.log 2024-09-18T00:42:29.920 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62869.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133124.log 2024-09-18T00:42:29.920 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69883.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.69883.log.gz 2024-09-18T00:42:29.920 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81057.log 2024-09-18T00:42:29.921 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68044.log 2024-09-18T00:42:29.922 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58271.log 2024-09-18T00:42:29.922 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68044.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.68044.log.gz 2024-09-18T00:42:29.922 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.83861.log.gz 2024-09-18T00:42:29.922 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115086.log 2024-09-18T00:42:29.923 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.58271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56101.log 2024-09-18T00:42:29.923 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115086.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.115086.log.gz 2024-09-18T00:42:29.924 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116615.log 2024-09-18T00:42:29.924 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56101.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.56101.log.gz 2024-09-18T00:42:29.924 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153769.log 2024-09-18T00:42:29.925 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.116615.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52091.log.gz 2024-09-18T00:42:29.925 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66060.log 2024-09-18T00:42:29.926 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.153769.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52140.log.gz 2024-09-18T00:42:29.926 INFO:teuthology.orchestra.run.smithi192.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.153769.log.gz 2024-09-18T00:42:29.926 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54609.log 2024-09-18T00:42:29.927 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66060.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52197.log.gz 2024-09-18T00:42:29.927 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106479.log 2024-09-18T00:42:29.927 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124609.log 2024-09-18T00:42:29.928 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.106479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43166.log 2024-09-18T00:42:29.929 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46749.log 2024-09-18T00:42:29.929 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43166.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127916.log 2024-09-18T00:42:29.930 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.46749.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36380.log 2024-09-18T00:42:29.930 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.127916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140328.log 2024-09-18T00:42:29.930 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118182.log 2024-09-18T00:42:29.931 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140328.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41792.log.gz 2024-09-18T00:42:29.931 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93625.log 2024-09-18T00:42:29.932 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.118182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136692.log 2024-09-18T00:42:29.932 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93625.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.93625.log.gz 2024-09-18T00:42:29.932 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123182.log 2024-09-18T00:42:29.933 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100039.log 2024-09-18T00:42:29.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123182.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111981.log.gz 2024-09-18T00:42:29.934 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135348.log 2024-09-18T00:42:29.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142840.log 2024-09-18T00:42:29.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104482.log 2024-09-18T00:42:29.936 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.142840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94655.log 2024-09-18T00:42:29.936 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.104482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151088.log 2024-09-18T00:42:29.936 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.119964.log.gz 2024-09-18T00:42:29.937 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.94655.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152702.log 2024-09-18T00:42:29.937 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151088.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.142383.log.gz 2024-09-18T00:42:29.938 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58821.log.gz 2024-09-18T00:42:29.938 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101049.log 2024-09-18T00:42:29.938 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152702.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.152702.log.gz 2024-09-18T00:42:29.939 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64027.log 2024-09-18T00:42:29.939 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101049.log: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70168.log.gz 2024-09-18T00:42:29.940 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121462.log 2024-09-18T00:42:29.940 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64027.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43373.log.gz 2024-09-18T00:42:29.940 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81584.log 2024-09-18T00:42:29.941 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121462.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.83045.log.gz 2024-09-18T00:42:29.941 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103813.log 2024-09-18T00:42:29.942 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81584.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81584.log.gz 2024-09-18T00:42:29.942 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121745.log 2024-09-18T00:42:29.942 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66734.log 2024-09-18T00:42:29.943 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121745.log: 91.2% -- replaced with /var/log/ceph/cluster1-client.admin.94199.log.gz 2024-09-18T00:42:29.943 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90348.log 2024-09-18T00:42:29.944 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66734.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145146.log 2024-09-18T00:42:29.945 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90348.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.104013.log.gz 2024-09-18T00:42:29.945 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111437.log 2024-09-18T00:42:29.945 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145146.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.142840.log.gz 2024-09-18T00:42:29.945 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121712.log 2024-09-18T00:42:29.946 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136064.log 2024-09-18T00:42:29.946 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72061.log 2024-09-18T00:42:29.947 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91411.log 2024-09-18T00:42:29.947 INFO:teuthology.orchestra.run.smithi192.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71474.log.gz 2024-09-18T00:42:29.948 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142798.log 2024-09-18T00:42:29.948 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52427.log.gz 2024-09-18T00:42:29.948 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91411.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91411.log.gz 2024-09-18T00:42:29.949 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121168.log 2024-09-18T00:42:29.950 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110382.log 2024-09-18T00:42:29.950 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89507.log 2024-09-18T00:42:29.950 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110382.log: gzip 89.4% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.94872.log.gz --verbose 2024-09-18T00:42:29.951 INFO:teuthology.orchestra.run.smithi192.stderr: -- /var/log/ceph/cluster2-client.admin.104882.log 2024-09-18T00:42:29.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133310.log 2024-09-18T00:42:29.952 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95013.log.gz 2024-09-18T00:42:29.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104882.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.95145.log.gz 2024-09-18T00:42:29.952 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90179.log 2024-09-18T00:42:29.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133310.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.104891.log.gz 2024-09-18T00:42:29.953 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112904.log 2024-09-18T00:42:29.953 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90179.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56262.log.gz 2024-09-18T00:42:29.953 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102433.log 2024-09-18T00:42:29.954 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122008.log 2024-09-18T00:42:29.955 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144805.log 2024-09-18T00:42:29.955 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122008.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141294.log 2024-09-18T00:42:29.955 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134485.log 2024-09-18T00:42:29.956 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63942.log 2024-09-18T00:42:29.956 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140708.log 2024-09-18T00:42:29.957 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54608.log.gz 2024-09-18T00:42:29.957 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63942.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54609.log.gz 2024-09-18T00:42:29.957 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148608.log 2024-09-18T00:42:29.958 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140708.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140708.log.gz 2024-09-18T00:42:29.958 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141838.log 2024-09-18T00:42:29.959 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138818.log 2024-09-18T00:42:29.959 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141838.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80822.log 2024-09-18T00:42:29.960 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129123.log 2024-09-18T00:42:29.960 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55197.log 2024-09-18T00:42:29.961 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129123.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71341.log 2024-09-18T00:42:29.962 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55197.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43282.log 2024-09-18T00:42:29.962 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.105514.log.gz 2024-09-18T00:42:29.962 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59907.log 2024-09-18T00:42:29.963 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43282.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58982.log 2024-09-18T00:42:29.963 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.59907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98990.log 2024-09-18T00:42:29.964 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.58982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126743.log 2024-09-18T00:42:29.964 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98990.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150544.log 2024-09-18T00:42:29.964 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.126743.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.126743.log.gz 2024-09-18T00:42:29.965 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110801.log 2024-09-18T00:42:29.965 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150544.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58754.log 2024-09-18T00:42:29.966 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68351.log 2024-09-18T00:42:29.966 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.58754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62995.log 2024-09-18T00:42:29.967 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133644.log 2024-09-18T00:42:29.967 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.104638.log.gz 2024-09-18T00:42:29.967 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59443.log 2024-09-18T00:42:29.968 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82959.log 2024-09-18T00:42:29.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106453.log 2024-09-18T00:42:29.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.82959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134781.log 2024-09-18T00:42:29.969 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.106453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.35608.log 2024-09-18T00:42:29.970 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36507.log 2024-09-18T00:42:29.970 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.0.35608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41182.log 2024-09-18T00:42:29.971 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46977.log 2024-09-18T00:42:29.971 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111351.log.gz 2024-09-18T00:42:29.972 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132736.log 2024-09-18T00:42:29.972 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138372.log.gz 2024-09-18T00:42:29.973 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.46977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129009.log 2024-09-18T00:42:29.973 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.132736.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52973.log 2024-09-18T00:42:29.973 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118094.log.gz 2024-09-18T00:42:29.973 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79596.log 2024-09-18T00:42:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124904.log 2024-09-18T00:42:29.974 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79596.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143192.log 2024-09-18T00:42:29.975 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49362.log 2024-09-18T00:42:29.975 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107373.log.gz 2024-09-18T00:42:29.976 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143192.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137530.log 2024-09-18T00:42:29.976 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42326.log 2024-09-18T00:42:29.977 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137530.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136608.log 2024-09-18T00:42:29.977 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146114.log 2024-09-18T00:42:29.978 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113116.log 2024-09-18T00:42:29.979 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146114.log: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.119460.log.gz 2024-09-18T00:42:29.979 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80626.log 2024-09-18T00:42:29.980 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113116.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.119496.log.gz 2024-09-18T00:42:29.980 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64799.log 2024-09-18T00:42:29.980 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.80626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122804.log 2024-09-18T00:42:29.981 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50270.log 2024-09-18T00:42:29.981 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43792.log 2024-09-18T00:42:29.982 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103272.log 2024-09-18T00:42:29.983 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137531.log 2024-09-18T00:42:29.983 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111221.log.gz 2024-09-18T00:42:29.983 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111222.log.gz 2024-09-18T00:42:29.983 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103272.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.103272.log.gz 2024-09-18T00:42:29.984 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66482.log 2024-09-18T00:42:29.984 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137531.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.82620.log.gz 2024-09-18T00:42:29.984 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111894.log 2024-09-18T00:42:29.985 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37427.log 2024-09-18T00:42:29.986 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90931.log 2024-09-18T00:42:29.986 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152398.log 2024-09-18T00:42:29.987 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44332.log 2024-09-18T00:42:29.987 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152398.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-18T00:42:29.988 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44332.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44332.log.gz 2024-09-18T00:42:29.988 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135534.log 2024-09-18T00:42:29.989 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39521.log 2024-09-18T00:42:29.989 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36471.log 2024-09-18T00:42:29.990 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78206.log 2024-09-18T00:42:29.991 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36471.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71039.log 2024-09-18T00:42:29.991 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78206.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136480.log 2024-09-18T00:42:29.992 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.71039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91564.log 2024-09-18T00:42:29.992 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136480.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107959.log 2024-09-18T00:42:29.993 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100206.log 2024-09-18T00:42:29.994 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89359.log.gz 2024-09-18T00:42:29.994 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109461.log 2024-09-18T00:42:29.995 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100206.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135808.log 2024-09-18T00:42:29.995 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90348.log.gz 2024-09-18T00:42:29.995 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109461.log: /var/log/ceph/cluster1-client.mirror.135808.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127877.log 2024-09-18T00:42:29.996 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106762.log 2024-09-18T00:42:29.996 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.90179.log.gz 2024-09-18T00:42:29.997 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127877.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81569.log 2024-09-18T00:42:29.997 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106762.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117793.log 2024-09-18T00:42:29.998 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81569.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81569.log.gz 2024-09-18T00:42:29.998 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65429.log 2024-09-18T00:42:29.998 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50659.log.gz 2024-09-18T00:42:29.999 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.117793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121923.log 2024-09-18T00:42:30.000 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65429.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100206.log.gz 2024-09-18T00:42:30.000 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141884.log 2024-09-18T00:42:30.001 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121923.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41372.log 2024-09-18T00:42:30.007 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141884.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101049.log.gz 2024-09-18T00:42:30.007 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116430.log 2024-09-18T00:42:30.007 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117699.log 2024-09-18T00:42:30.007 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82996.log 2024-09-18T00:42:30.007 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.117699.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.79278.log.gz 2024-09-18T00:42:30.007 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61419.log 2024-09-18T00:42:30.007 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.82996.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55829.log 2024-09-18T00:42:30.007 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.61419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134780.log 2024-09-18T00:42:30.007 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55829.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118000.log 2024-09-18T00:42:30.007 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134780.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75719.log 2024-09-18T00:42:30.007 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.118000.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43583.log 2024-09-18T00:42:30.007 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135700.log 2024-09-18T00:42:30.008 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142583.log 2024-09-18T00:42:30.009 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77995.log 2024-09-18T00:42:30.009 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66565.log 2024-09-18T00:42:30.009 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77995.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80448.log 2024-09-18T00:42:30.009 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40219.log 2024-09-18T00:42:30.010 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80448.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134720.log.gz 2024-09-18T00:42:30.010 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72216.log 2024-09-18T00:42:30.010 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72405.log 2024-09-18T00:42:30.011 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75088.log 2024-09-18T00:42:30.011 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72405.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137740.log 2024-09-18T00:42:30.012 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115577.log 2024-09-18T00:42:30.012 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117357.log 2024-09-18T00:42:30.013 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.115577.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121138.log 2024-09-18T00:42:30.013 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.117357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.151622.log 2024-09-18T00:42:30.014 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.151088.log.gz 2024-09-18T00:42:30.014 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65033.log 2024-09-18T00:42:30.014 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.151622.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.89507.log.gz 2024-09-18T00:42:30.015 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42323.log 2024-09-18T00:42:30.015 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65033.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140322.log 2024-09-18T00:42:30.016 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52256.log 2024-09-18T00:42:30.017 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54356.log 2024-09-18T00:42:30.017 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135114.log.gz 2024-09-18T00:42:30.017 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150599.log 2024-09-18T00:42:30.018 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54356.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43585.log 2024-09-18T00:42:30.019 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150599.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42115.log 2024-09-18T00:42:30.019 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43585.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66336.log.gz 2024-09-18T00:42:30.019 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74732.log 2024-09-18T00:42:30.020 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94085.log 2024-09-18T00:42:30.021 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89880.log 2024-09-18T00:42:30.021 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.94085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55322.log 2024-09-18T00:42:30.022 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133604.log 2024-09-18T00:42:30.022 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55322.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116610.log 2024-09-18T00:42:30.022 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64027.log.gz 2024-09-18T00:42:30.023 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82297.log 2024-09-18T00:42:30.024 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.116610.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68927.log 2024-09-18T00:42:30.024 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.82297.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85003.log 2024-09-18T00:42:30.025 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.68927.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116611.log 2024-09-18T00:42:30.026 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85003.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136064.log.gz 2024-09-18T00:42:30.026 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38577.log 2024-09-18T00:42:30.027 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.116611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58935.log 2024-09-18T00:42:30.027 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38577.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74733.log 2024-09-18T00:42:30.028 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.58935.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93231.log 2024-09-18T00:42:30.028 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74733.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.151693.log 2024-09-18T00:42:30.029 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109461.log.gz 2024-09-18T00:42:30.029 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.93231.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43374.log 2024-09-18T00:42:30.030 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.151693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45589.log 2024-09-18T00:42:30.030 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68141.log 2024-09-18T00:42:30.031 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.45589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81143.log 2024-09-18T00:42:30.031 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68141.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35300.log 2024-09-18T00:42:30.032 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44703.log 2024-09-18T00:42:30.032 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.35300.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48826.log 2024-09-18T00:42:30.033 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85892.log 2024-09-18T00:42:30.034 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129673.log 2024-09-18T00:42:30.035 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.56914.log.gz/var/log/ceph/cluster1-client.mirror.85892.log: 2024-09-18T00:42:30.035 INFO:teuthology.orchestra.run.smithi192.stderr: 86.5% -- replaced with /var/log/ceph/cluster2-client.admin.56905.log.gz 2024-09-18T00:42:30.035 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.56940.log.gz 2024-09-18T00:42:30.035 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78794.log 2024-09-18T00:42:30.035 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.129673.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57121.log.gz 2024-09-18T00:42:30.035 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57125.log.gz 2024-09-18T00:42:30.036 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34441.log 2024-09-18T00:42:30.036 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78794.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142771.log 2024-09-18T00:42:30.037 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.34441.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34441.log.gz 2024-09-18T00:42:30.037 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65136.log 2024-09-18T00:42:30.038 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83672.log 2024-09-18T00:42:30.038 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65136.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82095.log 2024-09-18T00:42:30.039 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83672.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46687.log 2024-09-18T00:42:30.040 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89151.log.gz 2024-09-18T00:42:30.040 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.82095.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129632.log 2024-09-18T00:42:30.041 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.46687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137783.log 2024-09-18T00:42:30.041 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129632.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49026.log 2024-09-18T00:42:30.041 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.94380.log.gz 2024-09-18T00:42:30.041 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137783.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152400.log 2024-09-18T00:42:30.042 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49026.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57639.log 2024-09-18T00:42:30.043 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119564.log 2024-09-18T00:42:30.044 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57639.log: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58479.log.gz 2024-09-18T00:42:30.044 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63834.log 2024-09-18T00:42:30.044 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51525.log.gz 2024-09-18T00:42:30.044 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.119564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48368.log 2024-09-18T00:42:30.045 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78312.log 2024-09-18T00:42:30.045 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48368.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62784.log 2024-09-18T00:42:30.046 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64004.log 2024-09-18T00:42:30.047 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62784.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148397.log 2024-09-18T00:42:30.047 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76196.log 2024-09-18T00:42:30.047 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148397.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58298.log 2024-09-18T00:42:30.048 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76196.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148397.log.gz 2024-09-18T00:42:30.048 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134590.log 2024-09-18T00:42:30.049 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.58298.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110591.log.gz 2024-09-18T00:42:30.049 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122049.log 2024-09-18T00:42:30.049 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134590.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42952.log 2024-09-18T00:42:30.050 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113112.log 2024-09-18T00:42:30.051 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42952.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109711.log.gz 2024-09-18T00:42:30.051 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36839.log 2024-09-18T00:42:30.051 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95862.log 2024-09-18T00:42:30.052 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135431.log 2024-09-18T00:42:30.052 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95862.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.106762.log.gz 2024-09-18T00:42:30.052 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77659.log 2024-09-18T00:42:30.053 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119155.log 2024-09-18T00:42:30.054 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115099.log 2024-09-18T00:42:30.054 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84039.log 2024-09-18T00:42:30.055 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.115099.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44371.log 2024-09-18T00:42:30.055 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65598.log 2024-09-18T00:42:30.056 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107875.log 2024-09-18T00:42:30.057 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65598.log: /var/log/ceph/cluster1-client.mirror.44371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136061.log 2024-09-18T00:42:30.057 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.107875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114166.log 2024-09-18T00:42:30.058 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66338.log 2024-09-18T00:42:30.058 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114166.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65767.log 2024-09-18T00:42:30.059 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66338.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147433.log 2024-09-18T00:42:30.059 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122409.log.gz 2024-09-18T00:42:30.060 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65767.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140695.log 2024-09-18T00:42:30.060 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147433.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.104882.log.gz 2024-09-18T00:42:30.060 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55136.log 2024-09-18T00:42:30.061 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140695.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140695.log.gz 2024-09-18T00:42:30.061 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115217.log 2024-09-18T00:42:30.062 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94892.log 2024-09-18T00:42:30.063 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.115217.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.105202.log.gz 2024-09-18T00:42:30.063 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135987.log 2024-09-18T00:42:30.065 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.94892.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137384.log 2024-09-18T00:42:30.065 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135987.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.admin.106453.log.gz 2024-09-18T00:42:30.065 INFO:teuthology.orchestra.run.smithi192.stderr:gzip 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106479.log.gz 2024-09-18T00:42:30.065 INFO:teuthology.orchestra.run.smithi192.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.104965.log 2024-09-18T00:42:30.065 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64738.log 2024-09-18T00:42:30.066 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121569.log 2024-09-18T00:42:30.066 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71340.log 2024-09-18T00:42:30.067 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121569.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119764.log 2024-09-18T00:42:30.067 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78146.log 2024-09-18T00:42:30.068 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119764.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140669.log 2024-09-18T00:42:30.069 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78146.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76945.log 2024-09-18T00:42:30.069 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140669.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.140669.log.gz 2024-09-18T00:42:30.070 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149261.log 2024-09-18T00:42:30.070 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76945.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55197.log.gz 2024-09-18T00:42:30.070 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55322.log.gz 2024-09-18T00:42:30.071 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153494.log 2024-09-18T00:42:30.072 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149261.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132455.log 2024-09-18T00:42:30.073 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.153494.log: 66.3% -- replaced with /var/log/ceph/cluster1-client.mirror.153494.log.gz 2024-09-18T00:42:30.073 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66146.log 2024-09-18T00:42:30.073 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.145146.log.gz 2024-09-18T00:42:30.073 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.132455.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44475.log 2024-09-18T00:42:30.074 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66146.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119039.log 2024-09-18T00:42:30.075 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44475.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101426.log 2024-09-18T00:42:30.075 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119039.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94587.log 2024-09-18T00:42:30.076 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101426.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42323.log.gz 2024-09-18T00:42:30.076 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149682.log 2024-09-18T00:42:30.077 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.94587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136125.log 2024-09-18T00:42:30.078 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149682.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48895.log 2024-09-18T00:42:30.078 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64991.log 2024-09-18T00:42:30.078 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.48895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124443.log 2024-09-18T00:42:30.079 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64991.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64506.log 2024-09-18T00:42:30.079 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72124.log 2024-09-18T00:42:30.080 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109882.log 2024-09-18T00:42:30.081 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109501.log 2024-09-18T00:42:30.081 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150379.log 2024-09-18T00:42:30.082 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109501.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.57137.log.gz 2024-09-18T00:42:30.083 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144427.log 2024-09-18T00:42:30.083 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150379.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.150379.log.gz 2024-09-18T00:42:30.083 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125744.log 2024-09-18T00:42:30.083 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144427.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.68351.log.gz 2024-09-18T00:42:30.084 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92939.log 2024-09-18T00:42:30.084 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125744.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90852.log 2024-09-18T00:42:30.085 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92939.log: 66.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58935.log.gz 2024-09-18T00:42:30.085 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102729.log 2024-09-18T00:42:30.086 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148035.log 2024-09-18T00:42:30.086 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116549.log 2024-09-18T00:42:30.087 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148035.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137804.log 2024-09-18T00:42:30.087 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.116549.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153541.log 2024-09-18T00:42:30.088 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142167.log 2024-09-18T00:42:30.088 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.148035.log.gz 2024-09-18T00:42:30.089 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.153541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54880.log 2024-09-18T00:42:30.089 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142167.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.admin.153541.log.gz 2024-09-18T00:42:30.089 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121923.log.gz 2024-09-18T00:42:30.089 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66481.log 2024-09-18T00:42:30.090 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145273.log 2024-09-18T00:42:30.091 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135495.log 2024-09-18T00:42:30.091 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145273.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112904.log.gz 2024-09-18T00:42:30.092 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66422.log 2024-09-18T00:42:30.092 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135495.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106111.log 2024-09-18T00:42:30.093 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66422.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96845.log.gz 2024-09-18T00:42:30.093 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53832.log 2024-09-18T00:42:30.094 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.106111.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100586.log 2024-09-18T00:42:30.094 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64843.log 2024-09-18T00:42:30.095 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100586.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92260.log 2024-09-18T00:42:30.096 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64843.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86519.log 2024-09-18T00:42:30.097 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135977.log 2024-09-18T00:42:30.097 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86519.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121462.log.gz 2024-09-18T00:42:30.097 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39632.log 2024-09-18T00:42:30.097 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108025.log 2024-09-18T00:42:30.098 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39632.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102138.log 2024-09-18T00:42:30.098 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43072.log 2024-09-18T00:42:30.099 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102138.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94339.log 2024-09-18T00:42:30.099 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96186.log 2024-09-18T00:42:30.099 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121138.log.gz 2024-09-18T00:42:30.100 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.94339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96015.log 2024-09-18T00:42:30.100 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96186.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121168.log.gz 2024-09-18T00:42:30.101 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59256.log 2024-09-18T00:42:30.101 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34903.log 2024-09-18T00:42:30.102 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.59256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126910.log 2024-09-18T00:42:30.102 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.34903.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34903.log.gz 2024-09-18T00:42:30.103 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43963.log 2024-09-18T00:42:30.103 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.126910.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.126910.log.gz 2024-09-18T00:42:30.103 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47434.log 2024-09-18T00:42:30.104 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65641.log 2024-09-18T00:42:30.104 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.47434.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99578.log 2024-09-18T00:42:30.105 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36317.log 2024-09-18T00:42:30.105 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99578.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67802.log 2024-09-18T00:42:30.106 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86224.log 2024-09-18T00:42:30.106 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.67802.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.67802.log.gz 2024-09-18T00:42:30.106 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107923.log 2024-09-18T00:42:30.107 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145226.log 2024-09-18T00:42:30.108 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107923.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95109.log 2024-09-18T00:42:30.108 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144849.log 2024-09-18T00:42:30.109 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95109.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112061.log 2024-09-18T00:42:30.109 INFO:teuthology.orchestra.run.smithi192.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-client.admin.141496.log.gz 2024-09-18T00:42:30.109 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125912.log 2024-09-18T00:42:30.110 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102936.log 2024-09-18T00:42:30.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54947.log 2024-09-18T00:42:30.111 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102936.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104256.log 2024-09-18T00:42:30.112 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54947.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127731.log 2024-09-18T00:42:30.112 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.104256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121628.log 2024-09-18T00:42:30.113 INFO:teuthology.orchestra.run.smithi192.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129673.log.gz 2024-09-18T00:42:30.113 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.127731.log: 88.1%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.129632.log.gz -5 2024-09-18T00:42:30.113 INFO:teuthology.orchestra.run.smithi192.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.133677.log 2024-09-18T00:42:30.114 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121628.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.81814.log 2024-09-18T00:42:30.115 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133677.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81241.log 2024-09-18T00:42:30.115 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.81814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53685.log 2024-09-18T00:42:30.116 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138160.log 2024-09-18T00:42:30.116 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90721.log 2024-09-18T00:42:30.117 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138160.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121627.log 2024-09-18T00:42:30.118 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90721.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125408.log 2024-09-18T00:42:30.118 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121712.log.gz 2024-09-18T00:42:30.118 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137110.log 2024-09-18T00:42:30.119 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125408.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114588.log 2024-09-18T00:42:30.120 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121205.log 2024-09-18T00:42:30.120 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114588.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59122.log 2024-09-18T00:42:30.121 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121205.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71403.log 2024-09-18T00:42:30.121 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.59122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86889.log 2024-09-18T00:42:30.122 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71403.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68708.log 2024-09-18T00:42:30.122 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86889.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89014.log 2024-09-18T00:42:30.123 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.68708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80576.log 2024-09-18T00:42:30.123 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145143.log 2024-09-18T00:42:30.124 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50321.log 2024-09-18T00:42:30.124 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126333.log 2024-09-18T00:42:30.125 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78944.log 2024-09-18T00:42:30.126 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56143.log 2024-09-18T00:42:30.126 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40321.log 2024-09-18T00:42:30.127 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.137804.log.gz 2024-09-18T00:42:30.127 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.56143.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45616.log 2024-09-18T00:42:30.128 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150185.log 2024-09-18T00:42:30.128 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.45616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.140899.log 2024-09-18T00:42:30.129 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150185.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104826.log 2024-09-18T00:42:30.129 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.140899.log: gzip 90.6% -5 --verbose -- replaced with /var/log/ceph/cluster1-client.mirror.50270.log.gz -- 2024-09-18T00:42:30.130 INFO:teuthology.orchestra.run.smithi192.stderr: /var/log/ceph/cluster1-client.mirror.134510.log 2024-09-18T00:42:30.130 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.114944.log 2024-09-18T00:42:30.131 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110388.log 2024-09-18T00:42:30.132 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.114944.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.114944.log.gz 2024-09-18T00:42:30.132 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45257.log 2024-09-18T00:42:30.132 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110388.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141583.log 2024-09-18T00:42:30.133 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66270.log 2024-09-18T00:42:30.133 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133605.log.gz 2024-09-18T00:42:30.134 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95749.log 2024-09-18T00:42:30.134 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66270.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57918.log 2024-09-18T00:42:30.135 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95749.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139742.log 2024-09-18T00:42:30.136 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72252.log 2024-09-18T00:42:30.136 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36380.log.gz 2024-09-18T00:42:30.136 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.139742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94427.log 2024-09-18T00:42:30.137 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143589.log 2024-09-18T00:42:30.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.94427.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136649.log 2024-09-18T00:42:30.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147791.log 2024-09-18T00:42:30.139 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.136480.log.gz 2024-09-18T00:42:30.139 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136649.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139027.log 2024-09-18T00:42:30.140 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147791.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93476.log 2024-09-18T00:42:30.141 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139027.log: 89.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85697.log 2024-09-18T00:42:30.141 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45589.log.gz 2024-09-18T00:42:30.141 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80178.log 2024-09-18T00:42:30.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85697.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112652.log 2024-09-18T00:42:30.142 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.80178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126954.log 2024-09-18T00:42:30.143 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67616.log 2024-09-18T00:42:30.144 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126954.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.85892.log.gz 2024-09-18T00:42:30.144 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56769.log 2024-09-18T00:42:30.145 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101217.log 2024-09-18T00:42:30.147 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40280.log 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77765.log.gz 2024-09-18T00:42:30.147 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:30.147 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101217.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56687.log 2024-09-18T00:42:30.147 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40280.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53139.log 2024-09-18T00:42:30.148 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51439.log 2024-09-18T00:42:30.148 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87477.log.gz 2024-09-18T00:42:30.148 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53139.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93769.log 2024-09-18T00:42:30.149 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51439.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134721.log 2024-09-18T00:42:30.149 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.93769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53768.log 2024-09-18T00:42:30.151 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128008.log 2024-09-18T00:42:30.151 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87274.log 2024-09-18T00:42:30.151 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1%/var/log/ceph/cluster2-client.admin.128008.log: -- replaced with /var/log/ceph/cluster1-client.mirror.76945.log.gz 2024-09-18T00:42:30.151 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137637.log 2024-09-18T00:42:30.152 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59755.log 2024-09-18T00:42:30.153 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74521.log 2024-09-18T00:42:30.153 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.59755.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76196.log.gz 2024-09-18T00:42:30.153 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91470.log 2024-09-18T00:42:30.154 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39864.log 2024-09-18T00:42:30.154 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91470.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91470.log.gz 2024-09-18T00:42:30.155 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86646.log 2024-09-18T00:42:30.155 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39864.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36839.log.gz 2024-09-18T00:42:30.155 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77681.log 2024-09-18T00:42:30.156 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86646.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98152.log 2024-09-18T00:42:30.157 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77681.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125913.log 2024-09-18T00:42:30.157 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52655.log 2024-09-18T00:42:30.158 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125913.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.152400.log.gz 2024-09-18T00:42:30.158 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68080.log 2024-09-18T00:42:30.159 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86225.log 2024-09-18T00:42:30.159 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68080.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126122.log 2024-09-18T00:42:30.160 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103600.log 2024-09-18T00:42:30.160 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65894.log 2024-09-18T00:42:30.161 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.151630.log 2024-09-18T00:42:30.162 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65894.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126333.log.gz 2024-09-18T00:42:30.162 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121244.log 2024-09-18T00:42:30.162 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.151630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82693.log 2024-09-18T00:42:30.163 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121244.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129445.log 2024-09-18T00:42:30.164 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.82693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74373.log 2024-09-18T00:42:30.164 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71136.log.gz 2024-09-18T00:42:30.164 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.129445.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49141.log 2024-09-18T00:42:30.165 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149006.log 2024-09-18T00:42:30.165 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49141.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40179.log 2024-09-18T00:42:30.166 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149006.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93174.log 2024-09-18T00:42:30.167 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102308.log 2024-09-18T00:42:30.168 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102686.log 2024-09-18T00:42:30.168 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119084.log 2024-09-18T00:42:30.168 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121205.log.gz 2024-09-18T00:42:30.169 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102686.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121244.log.gz 2024-09-18T00:42:30.169 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98907.log 2024-09-18T00:42:30.170 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119084.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69407.log 2024-09-18T00:42:30.171 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98907.log: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.115421.log.gz 2024-09-18T00:42:30.171 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131597.log 2024-09-18T00:42:30.172 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55576.log 2024-09-18T00:42:30.172 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131597.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78039.log 2024-09-18T00:42:30.173 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122259.log 2024-09-18T00:42:30.174 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.117357.log.gz 2024-09-18T00:42:30.174 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86643.log 2024-09-18T00:42:30.174 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122512.log 2024-09-18T00:42:30.175 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98526.log 2024-09-18T00:42:30.175 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73970.log 2024-09-18T00:42:30.175 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122343.log.gz 2024-09-18T00:42:30.175 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143333.log 2024-09-18T00:42:30.177 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90851.log 2024-09-18T00:42:30.177 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143333.log: 90.1%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.122049.log.gz 2024-09-18T00:42:30.177 INFO:teuthology.orchestra.run.smithi192.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.70101.log 2024-09-18T00:42:30.177 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46890.log 2024-09-18T00:42:30.178 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113744.log 2024-09-18T00:42:30.179 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.46890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95446.log 2024-09-18T00:42:30.179 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114166.log.gz 2024-09-18T00:42:30.179 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113744.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106137.log 2024-09-18T00:42:30.180 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95446.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65808.log 2024-09-18T00:42:30.181 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.106137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79090.log 2024-09-18T00:42:30.182 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65808.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112484.log 2024-09-18T00:42:30.182 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79090.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.95749.log.gz 2024-09-18T00:42:30.183 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64339.log 2024-09-18T00:42:30.183 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112484.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36271.log 2024-09-18T00:42:30.183 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120953.log 2024-09-18T00:42:30.184 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.36271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125138.log 2024-09-18T00:42:30.184 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105856.log 2024-09-18T00:42:30.185 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125138.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.139027.log.gz 2024-09-18T00:42:30.185 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151000.log 2024-09-18T00:42:30.186 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.105856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55951.log 2024-09-18T00:42:30.186 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95109.log.gz 2024-09-18T00:42:30.187 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151000.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104219.log 2024-09-18T00:42:30.187 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.104826.log.gz 2024-09-18T00:42:30.187 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.55951.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.55951.log.gz 2024-09-18T00:42:30.188 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4%gzip -- replaced with /var/log/ceph/cluster2-client.admin.92939.log.gz 2024-09-18T00:42:30.188 INFO:teuthology.orchestra.run.smithi192.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.150854.log 2024-09-18T00:42:30.190 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.104219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66881.log 2024-09-18T00:42:30.190 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150854.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47393.log 2024-09-18T00:42:30.191 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68791.log 2024-09-18T00:42:30.191 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.admin.94427.log.gz 2024-09-18T00:42:30.191 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.47393.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143475.log 2024-09-18T00:42:30.192 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.68791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66691.log 2024-09-18T00:42:30.192 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143475.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127874.log 2024-09-18T00:42:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112821.log 2024-09-18T00:42:30.194 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.127874.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112612.log.gz 2024-09-18T00:42:30.194 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41443.log 2024-09-18T00:42:30.194 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99367.log 2024-09-18T00:42:30.194 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41443.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112652.log.gz 2024-09-18T00:42:30.195 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134044.log 2024-09-18T00:42:30.196 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99367.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112694.log.gz 2024-09-18T00:42:30.196 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82571.log 2024-09-18T00:42:30.196 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112651.log 2024-09-18T00:42:30.197 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.82571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101512.log 2024-09-18T00:42:30.198 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54272.log 2024-09-18T00:42:30.198 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101512.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55829.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126251.log 2024-09-18T00:42:30.198 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:30.199 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78732.log 2024-09-18T00:42:30.200 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48851.log 2024-09-18T00:42:30.201 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91021.log 2024-09-18T00:42:30.201 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55576.log.gz 2024-09-18T00:42:30.201 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134110.log 2024-09-18T00:42:30.203 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133752.log 2024-09-18T00:42:30.203 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.94339.log.gz/var/log/ceph/cluster1-client.mirror.134110.log: 2024-09-18T00:42:30.203 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115217.log.gz 2024-09-18T00:42:30.203 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90899.log 2024-09-18T00:42:30.204 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.130207.log 2024-09-18T00:42:30.204 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45170.log 2024-09-18T00:42:30.205 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.130207.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44858.log 2024-09-18T00:42:30.205 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.45170.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93704.log 2024-09-18T00:42:30.206 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83429.log 2024-09-18T00:42:30.206 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93704.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.93704.log.gz 2024-09-18T00:42:30.207 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55787.log 2024-09-18T00:42:30.207 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83429.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53768.log.gz 2024-09-18T00:42:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40937.log 2024-09-18T00:42:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55787.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80511.log 2024-09-18T00:42:30.214 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40937.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.154277.log 2024-09-18T00:42:30.214 INFO:teuthology.orchestra.run.smithi192.stderr: 66.3% -- replaced with /var/log/ceph/cluster1-client.mirror.105836.log.gz 2024-09-18T00:42:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142297.log 2024-09-18T00:42:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.154277.log: 0.0% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106137.log.gz 2024-09-18T00:42:30.215 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.154277.log.gz 2024-09-18T00:42:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41375.log 2024-09-18T00:42:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142297.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.119039.log.gz 2024-09-18T00:42:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80974.log 2024-09-18T00:42:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120907.log 2024-09-18T00:42:30.215 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.119564.log.gz 2024-09-18T00:42:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115874.log 2024-09-18T00:42:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78460.log 2024-09-18T00:42:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.115874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43582.log 2024-09-18T00:42:30.216 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151465.log 2024-09-18T00:42:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65201.log 2024-09-18T00:42:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34357.log 2024-09-18T00:42:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124670.log 2024-09-18T00:42:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.34357.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34357.log.gz 2024-09-18T00:42:30.218 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153427.log 2024-09-18T00:42:30.218 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150599.log.gz 2024-09-18T00:42:30.218 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140972.log 2024-09-18T00:42:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.153427.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83602.log 2024-09-18T00:42:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.140972.log: 91.0% -- replaced with /var/log/ceph/cluster1-client.admin.153427.log.gz 2024-09-18T00:42:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121734.log 2024-09-18T00:42:30.220 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123248.log 2024-09-18T00:42:30.221 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121734.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145436.log 2024-09-18T00:42:30.221 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124609.log.gz 2024-09-18T00:42:30.221 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123248.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46707.log 2024-09-18T00:42:30.222 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145436.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44003.log 2024-09-18T00:42:30.223 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.46707.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149636.log 2024-09-18T00:42:30.223 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44003.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66881.log.gz 2024-09-18T00:42:30.223 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106292.log 2024-09-18T00:42:30.224 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149238.log 2024-09-18T00:42:30.225 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106292.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86183.log 2024-09-18T00:42:30.225 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149238.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91062.log 2024-09-18T00:42:30.226 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48436.log 2024-09-18T00:42:30.226 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58727.log 2024-09-18T00:42:30.227 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.48436.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136440.log 2024-09-18T00:42:30.227 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.58727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48179.log 2024-09-18T00:42:30.228 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147223.log 2024-09-18T00:42:30.229 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101302.log 2024-09-18T00:42:30.229 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137929.log 2024-09-18T00:42:30.230 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101302.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.147223.log.gz 2024-09-18T00:42:30.230 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140879.log 2024-09-18T00:42:30.230 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137929.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.66481.log.gz 2024-09-18T00:42:30.231 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30565.log 2024-09-18T00:42:30.232 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.140879.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71116.log 2024-09-18T00:42:30.232 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66565.log.gz 2024-09-18T00:42:30.232 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.30565.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30565.log.gz 2024-09-18T00:42:30.232 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51414.log 2024-09-18T00:42:30.233 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58132.log 2024-09-18T00:42:30.234 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51414.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122762.log 2024-09-18T00:42:30.234 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.58132.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.58132.log.gz 2024-09-18T00:42:30.234 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71059.log 2024-09-18T00:42:30.235 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122762.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66691.log.gz 2024-09-18T00:42:30.236 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75423.log 2024-09-18T00:42:30.236 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.71059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124652.log 2024-09-18T00:42:30.237 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88788.log 2024-09-18T00:42:30.237 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124652.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137531.log.gz 2024-09-18T00:42:30.238 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84391.log 2024-09-18T00:42:30.238 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88788.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120842.log 2024-09-18T00:42:30.239 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70261.log 2024-09-18T00:42:30.239 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147264.log 2024-09-18T00:42:30.240 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70261.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130427.log 2024-09-18T00:42:30.240 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135977.log.gz 2024-09-18T00:42:30.241 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.135987.log.gz 2024-09-18T00:42:30.241 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147264.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153608.log 2024-09-18T00:42:30.241 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130427.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.147264.log.gz 2024-09-18T00:42:30.242 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.152714.log 2024-09-18T00:42:30.242 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.153608.log: 66.3% -- replaced with /var/log/ceph/cluster1-client.mirror.153608.log.gz 2024-09-18T00:42:30.242 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52172.log 2024-09-18T00:42:30.243 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.152714.log: 90.0% 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.144427.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.152714.log.gz 2024-09-18T00:42:30.243 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:30.243 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53831.log 2024-09-18T00:42:30.244 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142886.log 2024-09-18T00:42:30.245 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96140.log 2024-09-18T00:42:30.245 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88959.log 2024-09-18T00:42:30.246 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.96140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56030.log 2024-09-18T00:42:30.247 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152624.log 2024-09-18T00:42:30.248 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.56030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38948.log 2024-09-18T00:42:30.248 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152624.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.152624.log.gz 2024-09-18T00:42:30.248 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92356.log 2024-09-18T00:42:30.249 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38948.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142971.log 2024-09-18T00:42:30.249 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92356.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.132455.log.gz 2024-09-18T00:42:30.250 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128694.log 2024-09-18T00:42:30.250 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67491.log 2024-09-18T00:42:30.250 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128694.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117726.log 2024-09-18T00:42:30.251 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153815.log 2024-09-18T00:42:30.252 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.117726.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70013.log 2024-09-18T00:42:30.252 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.153815.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87700.log.gz 2024-09-18T00:42:30.252 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143039.log 2024-09-18T00:42:30.253 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112272.log 2024-09-18T00:42:30.253 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96293.log 2024-09-18T00:42:30.254 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122914.log 2024-09-18T00:42:30.254 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.96293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115602.log 2024-09-18T00:42:30.255 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122914.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123056.log 2024-09-18T00:42:30.256 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138628.log 2024-09-18T00:42:30.256 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85820.log 2024-09-18T00:42:30.257 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.138628.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.138628.log.gz 2024-09-18T00:42:30.257 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47071.log 2024-09-18T00:42:30.258 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83670.log 2024-09-18T00:42:30.258 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47071.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45616.log.gz 2024-09-18T00:42:30.258 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.153815.log.gz 2024-09-18T00:42:30.258 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.93830.log 2024-09-18T00:42:30.259 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123792.log 2024-09-18T00:42:30.259 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.93830.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74268.log 2024-09-18T00:42:30.260 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74918.log 2024-09-18T00:42:30.261 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67940.log 2024-09-18T00:42:30.261 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.132736.log.gz 2024-09-18T00:42:30.262 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41653.log 2024-09-18T00:42:30.262 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149873.log 2024-09-18T00:42:30.263 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107196.log 2024-09-18T00:42:30.263 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149873.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145273.log.gz 2024-09-18T00:42:30.264 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76844.log 2024-09-18T00:42:30.264 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107196.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125913.log.gz 2024-09-18T00:42:30.264 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72168.log 2024-09-18T00:42:30.265 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44523.log 2024-09-18T00:42:30.266 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.72168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144724.log 2024-09-18T00:42:30.266 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44523.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44523.log.gz 2024-09-18T00:42:30.266 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77324.log 2024-09-18T00:42:30.267 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98883.log 2024-09-18T00:42:30.268 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138160.log.gz 2024-09-18T00:42:30.268 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88310.log 2024-09-18T00:42:30.268 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98883.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80627.log 2024-09-18T00:42:30.269 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110552.log 2024-09-18T00:42:30.269 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80627.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133752.log.gz 2024-09-18T00:42:30.270 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108226.log 2024-09-18T00:42:30.270 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51038.log 2024-09-18T00:42:30.271 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79342.log 2024-09-18T00:42:30.272 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51038.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87762.log 2024-09-18T00:42:30.272 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79342.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123604.log 2024-09-18T00:42:30.273 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87762.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.137637.log.gz 2024-09-18T00:42:30.273 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106613.log 2024-09-18T00:42:30.274 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123604.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.151895.log 2024-09-18T00:42:30.274 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.106613.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69126.log 2024-09-18T00:42:30.275 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.151895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105090.log 2024-09-18T00:42:30.275 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69126.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.85820.log.gz 2024-09-18T00:42:30.276 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52593.log 2024-09-18T00:42:30.276 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.105090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107547.log 2024-09-18T00:42:30.277 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52593.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89739.log.gz 2024-09-18T00:42:30.277 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95247.log 2024-09-18T00:42:30.278 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72277.log 2024-09-18T00:42:30.278 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136019.log 2024-09-18T00:42:30.279 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72277.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138140.log 2024-09-18T00:42:30.279 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136019.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139695.log 2024-09-18T00:42:30.280 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138140.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.76339.log.gz 2024-09-18T00:42:30.280 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124150.log 2024-09-18T00:42:30.281 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139695.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70033.log 2024-09-18T00:42:30.281 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124150.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68460.log 2024-09-18T00:42:30.282 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146173.log 2024-09-18T00:42:30.282 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66398.log 2024-09-18T00:42:30.284 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112907.log 2024-09-18T00:42:30.284 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66398.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76525.log.gz 2024-09-18T00:42:30.284 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88516.log 2024-09-18T00:42:30.284 INFO:teuthology.orchestra.run.smithi192.stderr: 19.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.35608.log.gz 2024-09-18T00:42:30.284 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112907.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146173.log.gz 2024-09-18T00:42:30.284 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43162.log 2024-09-18T00:42:30.285 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146845.log 2024-09-18T00:42:30.285 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43162.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94152.log 2024-09-18T00:42:30.286 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146845.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67320.log 2024-09-18T00:42:30.287 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.94152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60690.log 2024-09-18T00:42:30.287 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.146845.log.gz 2024-09-18T00:42:30.287 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.98907.log.gz 2024-09-18T00:42:30.288 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126938.log 2024-09-18T00:42:30.289 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.60690.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102308.log.gz 2024-09-18T00:42:30.289 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131300.log 2024-09-18T00:42:30.289 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.126938.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.126938.log.gz 2024-09-18T00:42:30.290 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54818.log 2024-09-18T00:42:30.290 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131300.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75574.log 2024-09-18T00:42:30.291 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95772.log 2024-09-18T00:42:30.292 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141221.log 2024-09-18T00:42:30.292 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.86183.log.gz 2024-09-18T00:42:30.292 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68787.log 2024-09-18T00:42:30.293 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.141221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146216.log 2024-09-18T00:42:30.293 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.68787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104179.log 2024-09-18T00:42:30.294 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67154.log 2024-09-18T00:42:30.295 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.151000.log.gz 2024-09-18T00:42:30.295 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.104179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39129.log 2024-09-18T00:42:30.296 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67154.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151019.log 2024-09-18T00:42:30.296 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39129.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.151895.log.gz 2024-09-18T00:42:30.296 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115603.log 2024-09-18T00:42:30.296 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86957.log.gz 2024-09-18T00:42:30.297 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89651.log 2024-09-18T00:42:30.297 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75672.log 2024-09-18T00:42:30.298 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112908.log 2024-09-18T00:42:30.298 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.140879.log.gz 2024-09-18T00:42:30.298 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75672.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101466.log.gz 2024-09-18T00:42:30.299 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115912.log 2024-09-18T00:42:30.299 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112908.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.112651.log.gz 2024-09-18T00:42:30.300 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113281.log 2024-09-18T00:42:30.301 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.115912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142644.log 2024-09-18T00:42:30.301 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113281.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101097.log 2024-09-18T00:42:30.302 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51878.log 2024-09-18T00:42:30.302 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101097.log: 90.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41374.log 2024-09-18T00:42:30.302 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.63540.log.gz 2024-09-18T00:42:30.304 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51878.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75463.log 2024-09-18T00:42:30.304 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63583.log.gz 2024-09-18T00:42:30.304 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41374.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63585.log.gz 2024-09-18T00:42:30.304 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63625.log.gz 2024-09-18T00:42:30.304 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143230.log 2024-09-18T00:42:30.305 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75463.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52172.log.gz 2024-09-18T00:42:30.305 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98567.log 2024-09-18T00:42:30.305 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143230.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.47434.log.gz 2024-09-18T00:42:30.306 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117498.log 2024-09-18T00:42:30.306 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87207.log 2024-09-18T00:42:30.307 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.117498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59296.log 2024-09-18T00:42:30.308 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87207.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52340.log 2024-09-18T00:42:30.308 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59296.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139797.log 2024-09-18T00:42:30.309 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131153.log 2024-09-18T00:42:30.309 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139797.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148001.log 2024-09-18T00:42:30.310 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100628.log 2024-09-18T00:42:30.311 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93924.log 2024-09-18T00:42:30.311 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92825.log 2024-09-18T00:42:30.311 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40010.log.gz 2024-09-18T00:42:30.312 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.93924.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40853.log 2024-09-18T00:42:30.312 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92825.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64381.log 2024-09-18T00:42:30.313 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112696.log 2024-09-18T00:42:30.314 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64381.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.94914.log.gz 2024-09-18T00:42:30.314 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91487.log 2024-09-18T00:42:30.314 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82323.log 2024-09-18T00:42:30.314 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91487.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91487.log.gz 2024-09-18T00:42:30.315 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59313.log 2024-09-18T00:42:30.316 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.82323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138773.log 2024-09-18T00:42:30.317 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134424.log 2024-09-18T00:42:30.317 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.138773.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65429.log.gz 2024-09-18T00:42:30.317 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.138773.log.gz 2024-09-18T00:42:30.317 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79899.log 2024-09-18T00:42:30.318 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59537.log 2024-09-18T00:42:30.318 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153058.log 2024-09-18T00:42:30.319 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.82736.log.gz 2024-09-18T00:42:30.319 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59537.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64128.log.gz 2024-09-18T00:42:30.319 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121094.log 2024-09-18T00:42:30.320 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.153058.log: 87.3%gzip -- replaced with /var/log/ceph/cluster1-client.admin.153058.log.gz 2024-09-18T00:42:30.320 INFO:teuthology.orchestra.run.smithi192.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.admin.45999.log 2024-09-18T00:42:30.321 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78270.log 2024-09-18T00:42:30.321 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151728.log 2024-09-18T00:42:30.322 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104182.log 2024-09-18T00:42:30.322 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39359.log.gz 2024-09-18T00:42:30.322 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.151728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121568.log 2024-09-18T00:42:30.323 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.104182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102560.log 2024-09-18T00:42:30.324 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121568.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82551.log 2024-09-18T00:42:30.324 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102560.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142924.log 2024-09-18T00:42:30.325 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.82551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75320.log 2024-09-18T00:42:30.326 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142924.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89129.log 2024-09-18T00:42:30.326 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136398.log 2024-09-18T00:42:30.327 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89129.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133753.log 2024-09-18T00:42:30.327 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66060.log.gz 2024-09-18T00:42:30.327 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131020.log 2024-09-18T00:42:30.328 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133753.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34083.log 2024-09-18T00:42:30.329 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122702.log 2024-09-18T00:42:30.329 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.34083.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34083.log.gz 2024-09-18T00:42:30.330 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148818.log 2024-09-18T00:42:30.330 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122702.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124670.log.gz 2024-09-18T00:42:30.330 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94038.log 2024-09-18T00:42:30.331 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38136.log 2024-09-18T00:42:30.331 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.94038.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79900.log 2024-09-18T00:42:30.332 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38136.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125138.log.gz 2024-09-18T00:42:30.332 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66882.log 2024-09-18T00:42:30.333 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79900.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67868.log 2024-09-18T00:42:30.333 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107221.log 2024-09-18T00:42:30.334 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.67868.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.67868.log.gz 2024-09-18T00:42:30.334 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115157.log 2024-09-18T00:42:30.335 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51499.log 2024-09-18T00:42:30.335 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.115157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136648.log 2024-09-18T00:42:30.336 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51499.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.110176.log.gz 2024-09-18T00:42:30.336 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91541.log 2024-09-18T00:42:30.337 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39634.log 2024-09-18T00:42:30.337 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91541.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91541.log.gz 2024-09-18T00:42:30.337 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122912.log 2024-09-18T00:42:30.338 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39634.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68955.log 2024-09-18T00:42:30.339 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87555.log 2024-09-18T00:42:30.339 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.68955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89462.log 2024-09-18T00:42:30.340 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87555.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112277.log 2024-09-18T00:42:30.340 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89462.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57471.log 2024-09-18T00:42:30.340 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112277.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116992.log 2024-09-18T00:42:30.341 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133437.log 2024-09-18T00:42:30.342 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.116992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134173.log 2024-09-18T00:42:30.342 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133437.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69389.log 2024-09-18T00:42:30.343 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41443.log.gz 2024-09-18T00:42:30.343 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58412.log 2024-09-18T00:42:30.344 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61174.log 2024-09-18T00:42:30.344 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.58412.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122512.log.gz 2024-09-18T00:42:30.344 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145484.log 2024-09-18T00:42:30.345 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.61174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109712.log 2024-09-18T00:42:30.345 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145484.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125828.log 2024-09-18T00:42:30.346 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.131153.log.gz 2024-09-18T00:42:30.346 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.80511.log.gz 2024-09-18T00:42:30.346 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147894.log 2024-09-18T00:42:30.347 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109712.log: /var/log/ceph/cluster1-client.mirror.125828.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55808.log 2024-09-18T00:42:30.348 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35368.log 2024-09-18T00:42:30.348 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55808.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.80974.log.gz 2024-09-18T00:42:30.348 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.94038.log.gz 2024-09-18T00:42:30.348 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145333.log 2024-09-18T00:42:30.348 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.94152.log.gz 2024-09-18T00:42:30.349 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.35368.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44103.log 2024-09-18T00:42:30.350 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137950.log 2024-09-18T00:42:30.350 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94225.log.gz 2024-09-18T00:42:30.351 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141610.log 2024-09-18T00:42:30.351 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.60690.log.gz 2024-09-18T00:42:30.351 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119521.log 2024-09-18T00:42:30.352 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144218.log 2024-09-18T00:42:30.352 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89951.log 2024-09-18T00:42:30.353 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99020.log 2024-09-18T00:42:30.353 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48873.log 2024-09-18T00:42:30.354 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99020.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.151906.log 2024-09-18T00:42:30.355 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47698.log 2024-09-18T00:42:30.355 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.151906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89927.log 2024-09-18T00:42:30.356 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113032.log 2024-09-18T00:42:30.357 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145586.log 2024-09-18T00:42:30.358 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113032.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.admin.118296.log.gz 2024-09-18T00:42:30.358 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146636.log 2024-09-18T00:42:30.358 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145586.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74879.log 2024-09-18T00:42:30.358 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141154.log 2024-09-18T00:42:30.358 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74879.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64299.log 2024-09-18T00:42:30.359 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141154.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76821.log 2024-09-18T00:42:30.359 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112231.log 2024-09-18T00:42:30.359 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76821.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103425.log.gz 2024-09-18T00:42:30.359 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41603.log 2024-09-18T00:42:30.360 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112231.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.103587.log.gz 2024-09-18T00:42:30.360 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.103600.log.gz 2024-09-18T00:42:30.360 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152650.log 2024-09-18T00:42:30.361 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44610.log 2024-09-18T00:42:30.361 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152650.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.152650.log.gz 2024-09-18T00:42:30.362 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81753.log 2024-09-18T00:42:30.362 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44610.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44610.log.gz 2024-09-18T00:42:30.363 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66564.log.gz 2024-09-18T00:42:30.363 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80499.log 2024-09-18T00:42:30.363 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.81753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55072.log 2024-09-18T00:42:30.364 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146088.log 2024-09-18T00:42:30.364 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123792.log.gz 2024-09-18T00:42:30.364 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80035.log 2024-09-18T00:42:30.365 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.146088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86310.log 2024-09-18T00:42:30.366 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.80035.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36927.log 2024-09-18T00:42:30.366 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86310.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112907.log.gz 2024-09-18T00:42:30.366 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106064.log 2024-09-18T00:42:30.367 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110386.log 2024-09-18T00:42:30.368 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.106064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36355.log 2024-09-18T00:42:30.368 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.92260.log.gz 2024-09-18T00:42:30.369 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110386.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55743.log 2024-09-18T00:42:30.369 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115368.log 2024-09-18T00:42:30.370 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55743.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.96293.log.gz 2024-09-18T00:42:30.370 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128454.log 2024-09-18T00:42:30.371 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115368.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46009.log 2024-09-18T00:42:30.371 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40430.log 2024-09-18T00:42:30.372 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46009.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58041.log 2024-09-18T00:42:30.372 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.56487.log.gz 2024-09-18T00:42:30.373 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136419.log 2024-09-18T00:42:30.373 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.58041.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.58041.log.gz 2024-09-18T00:42:30.373 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90092.log 2024-09-18T00:42:30.374 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144726.log 2024-09-18T00:42:30.374 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41695.log 2024-09-18T00:42:30.375 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144726.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59537.log.gz 2024-09-18T00:42:30.375 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47487.log 2024-09-18T00:42:30.376 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41695.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134530.log 2024-09-18T00:42:30.377 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.47487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83556.log 2024-09-18T00:42:30.377 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134530.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83518.log 2024-09-18T00:42:30.378 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83556.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56635.log 2024-09-18T00:42:30.378 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83518.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112821.log.gz 2024-09-18T00:42:30.378 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77576.log 2024-09-18T00:42:30.379 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143648.log 2024-09-18T00:42:30.380 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40089.log 2024-09-18T00:42:30.380 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149113.log 2024-09-18T00:42:30.381 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54482.log 2024-09-18T00:42:30.382 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118533.log 2024-09-18T00:42:30.382 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129834.log 2024-09-18T00:42:30.383 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118533.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59236.log 2024-09-18T00:42:30.383 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75672.log.gz 2024-09-18T00:42:30.384 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42534.log 2024-09-18T00:42:30.384 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.59236.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98444.log 2024-09-18T00:42:30.385 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40932.log 2024-09-18T00:42:30.386 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98444.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66773.log 2024-09-18T00:42:30.386 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118932.log 2024-09-18T00:42:30.387 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66773.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149658.log 2024-09-18T00:42:30.387 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144426.log 2024-09-18T00:42:30.388 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149658.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89817.log 2024-09-18T00:42:30.388 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.73970.log.gz 2024-09-18T00:42:30.389 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139970.log 2024-09-18T00:42:30.389 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147478.log 2024-09-18T00:42:30.390 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129807.log 2024-09-18T00:42:30.390 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147478.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41374.log.gz 2024-09-18T00:42:30.391 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55340.log 2024-09-18T00:42:30.391 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129807.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.147478.log.gz 2024-09-18T00:42:30.392 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82183.log 2024-09-18T00:42:30.392 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71647.log 2024-09-18T00:42:30.393 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.82183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42422.log 2024-09-18T00:42:30.394 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63833.log 2024-09-18T00:42:30.394 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.145484.log.gz 2024-09-18T00:42:30.394 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42422.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109921.log 2024-09-18T00:42:30.394 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63833.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.143405.log.gz 2024-09-18T00:42:30.395 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76715.log 2024-09-18T00:42:30.396 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109921.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147643.log 2024-09-18T00:42:30.396 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134697.log 2024-09-18T00:42:30.397 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153587.log 2024-09-18T00:42:30.397 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134697.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130165.log 2024-09-18T00:42:30.398 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.153587.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.147643.log.gz 2024-09-18T00:42:30.398 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59142.log 2024-09-18T00:42:30.399 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110805.log 2024-09-18T00:42:30.399 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.59142.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87955.log.gz 2024-09-18T00:42:30.400 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99957.log 2024-09-18T00:42:30.400 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70510.log 2024-09-18T00:42:30.400 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99957.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117524.log 2024-09-18T00:42:30.401 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.70510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75360.log 2024-09-18T00:42:30.402 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.117524.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90221.log 2024-09-18T00:42:30.402 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90092.log.gz 2024-09-18T00:42:30.403 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113115.log 2024-09-18T00:42:30.403 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124508.log 2024-09-18T00:42:30.404 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113115.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.153587.log.gz 2024-09-18T00:42:30.404 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149364.log 2024-09-18T00:42:30.405 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124508.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121204.log 2024-09-18T00:42:30.405 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88310.log.gz 2024-09-18T00:42:30.405 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42743.log 2024-09-18T00:42:30.406 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34525.log 2024-09-18T00:42:30.407 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36319.log 2024-09-18T00:42:30.407 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.34525.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34525.log.gz 2024-09-18T00:42:30.407 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78270.log.gz 2024-09-18T00:42:30.408 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.138978.log 2024-09-18T00:42:30.409 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36319.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.104957.log 2024-09-18T00:42:30.410 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.138978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143103.log 2024-09-18T00:42:30.410 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.46009.log.gz 2024-09-18T00:42:30.410 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.104957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68443.log 2024-09-18T00:42:30.411 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123249.log 2024-09-18T00:42:30.412 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92005.log 2024-09-18T00:42:30.412 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123249.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.151019.log.gz 2024-09-18T00:42:30.412 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72856.log 2024-09-18T00:42:30.412 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51414.log.gz 2024-09-18T00:42:30.413 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92005.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93652.log 2024-09-18T00:42:30.413 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72856.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.152043.log.gz 2024-09-18T00:42:30.414 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79132.log 2024-09-18T00:42:30.414 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93652.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.93652.log.gz 2024-09-18T00:42:30.414 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88249.log 2024-09-18T00:42:30.415 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145333.log.gz 2024-09-18T00:42:30.415 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81348.log 2024-09-18T00:42:30.416 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88249.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101427.log 2024-09-18T00:42:30.416 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81348.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94058.log 2024-09-18T00:42:30.417 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117252.log 2024-09-18T00:42:30.417 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109712.log.gz 2024-09-18T00:42:30.418 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.94058.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122594.log 2024-09-18T00:42:30.418 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.117252.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.117252.log.gz 2024-09-18T00:42:30.419 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46573.log 2024-09-18T00:42:30.420 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122594.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112487.log 2024-09-18T00:42:30.420 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.46573.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.127808.log 2024-09-18T00:42:30.421 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112487.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106043.log 2024-09-18T00:42:30.421 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.127808.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137871.log 2024-09-18T00:42:30.422 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.106043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75823.log 2024-09-18T00:42:30.423 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137871.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110386.log.gz 2024-09-18T00:42:30.423 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101570.log 2024-09-18T00:42:30.423 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41812.log.gz 2024-09-18T00:42:30.423 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40131.log 2024-09-18T00:42:30.424 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101570.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129746.log 2024-09-18T00:42:30.425 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36223.log 2024-09-18T00:42:30.426 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89382.log 2024-09-18T00:42:30.426 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101097.log.gz 2024-09-18T00:42:30.426 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89838.log 2024-09-18T00:42:30.427 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98487.log 2024-09-18T00:42:30.427 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102138.log.gz 2024-09-18T00:42:30.428 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137741.log 2024-09-18T00:42:30.428 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123582.log 2024-09-18T00:42:30.429 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125558.log 2024-09-18T00:42:30.429 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123582.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.154403.log 2024-09-18T00:42:30.430 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126607.log 2024-09-18T00:42:30.431 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.154403.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62910.log.gz 2024-09-18T00:42:30.431 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.154403.log.gz 2024-09-18T00:42:30.431 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100883.log 2024-09-18T00:42:30.431 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126607.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69857.log 2024-09-18T00:42:30.433 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126778.log 2024-09-18T00:42:30.433 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62994.log.gz 2024-09-18T00:42:30.433 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69857.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.69857.log.gz 2024-09-18T00:42:30.433 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51165.log 2024-09-18T00:42:30.433 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.126778.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.126778.log.gz 2024-09-18T00:42:30.434 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58096.log 2024-09-18T00:42:30.435 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96519.log 2024-09-18T00:42:30.435 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.58096.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.58096.log.gz 2024-09-18T00:42:30.435 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113321.log 2024-09-18T00:42:30.436 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109509.log 2024-09-18T00:42:30.436 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45105.log 2024-09-18T00:42:30.437 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51438.log 2024-09-18T00:42:30.437 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.45105.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35798.log 2024-09-18T00:42:30.438 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51438.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148737.log 2024-09-18T00:42:30.439 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.35798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59333.log 2024-09-18T00:42:30.439 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148737.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.116992.log.gz 2024-09-18T00:42:30.440 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76963.log 2024-09-18T00:42:30.440 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.59333.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74623.log.gz 2024-09-18T00:42:30.440 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87869.log 2024-09-18T00:42:30.441 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123643.log 2024-09-18T00:42:30.441 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87869.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55280.log 2024-09-18T00:42:30.442 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61418.log 2024-09-18T00:42:30.442 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55280.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100649.log 2024-09-18T00:42:30.443 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131020.log.gz 2024-09-18T00:42:30.443 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.61418.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35334.log 2024-09-18T00:42:30.444 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100649.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126978.log 2024-09-18T00:42:30.445 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.35334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85285.log 2024-09-18T00:42:30.445 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.126978.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.126978.log.gz 2024-09-18T00:42:30.445 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51437.log 2024-09-18T00:42:30.445 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85285.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38948.log.gz 2024-09-18T00:42:30.446 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36187.log 2024-09-18T00:42:30.447 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50474.log 2024-09-18T00:42:30.447 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88059.log 2024-09-18T00:42:30.448 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50474.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75320.log.gz 2024-09-18T00:42:30.448 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104932.log 2024-09-18T00:42:30.449 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30731.log 2024-09-18T00:42:30.449 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104932.log: 53.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35368.log.gz 2024-09-18T00:42:30.450 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131479.log 2024-09-18T00:42:30.450 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.30731.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30731.log.gz 2024-09-18T00:42:30.450 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75423.log.gz 2024-09-18T00:42:30.450 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75463.log.gz 2024-09-18T00:42:30.451 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65390.log 2024-09-18T00:42:30.452 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124066.log 2024-09-18T00:42:30.452 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112906.log 2024-09-18T00:42:30.453 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124066.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.144724.log.gz 2024-09-18T00:42:30.453 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63246.log 2024-09-18T00:42:30.454 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112906.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105502.log 2024-09-18T00:42:30.454 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63246.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83624.log 2024-09-18T00:42:30.455 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.105502.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105502.log.gz 2024-09-18T00:42:30.455 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100077.log 2024-09-18T00:42:30.456 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.83624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122258.log 2024-09-18T00:42:30.456 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100077.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51038.log.gz 2024-09-18T00:42:30.457 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125560.log 2024-09-18T00:42:30.457 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149994.log 2024-09-18T00:42:30.458 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125560.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117265.log 2024-09-18T00:42:30.458 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149994.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62197.log 2024-09-18T00:42:30.459 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.117265.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.117265.log.gz 2024-09-18T00:42:30.459 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72291.log 2024-09-18T00:42:30.460 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79546.log.gz 2024-09-18T00:42:30.460 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62197.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79843.log 2024-09-18T00:42:30.461 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116380.log 2024-09-18T00:42:30.461 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79843.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113957.log 2024-09-18T00:42:30.462 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145353.log 2024-09-18T00:42:30.463 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90014.log 2024-09-18T00:42:30.463 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73300.log 2024-09-18T00:42:30.464 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139249.log 2024-09-18T00:42:30.464 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73300.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60566.log 2024-09-18T00:42:30.465 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139249.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35556.log 2024-09-18T00:42:30.465 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60566.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122009.log.gz 2024-09-18T00:42:30.466 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139929.log 2024-09-18T00:42:30.466 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.35556.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85821.log 2024-09-18T00:42:30.467 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.81348.log.gz 2024-09-18T00:42:30.467 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.139929.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54651.log 2024-09-18T00:42:30.468 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64717.log 2024-09-18T00:42:30.468 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51586.log 2024-09-18T00:42:30.469 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122703.log 2024-09-18T00:42:30.469 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51586.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42422.log.gz 2024-09-18T00:42:30.470 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49232.log 2024-09-18T00:42:30.470 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67239.log 2024-09-18T00:42:30.471 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.49232.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42743.log.gz 2024-09-18T00:42:30.471 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98676.log 2024-09-18T00:42:30.472 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67239.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128132.log 2024-09-18T00:42:30.472 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125850.log 2024-09-18T00:42:30.473 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62281.log 2024-09-18T00:42:30.474 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71013.log 2024-09-18T00:42:30.474 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112231.log.gz 2024-09-18T00:42:30.474 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62281.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67406.log 2024-09-18T00:42:30.475 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.71013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139040.log 2024-09-18T00:42:30.476 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67406.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95925.log 2024-09-18T00:42:30.477 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113031.log 2024-09-18T00:42:30.477 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95925.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126293.log 2024-09-18T00:42:30.478 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66104.log 2024-09-18T00:42:30.478 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126375.log 2024-09-18T00:42:30.479 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66104.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65136.log.gz 2024-09-18T00:42:30.479 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83298.log 2024-09-18T00:42:30.480 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126375.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57478.log 2024-09-18T00:42:30.480 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141107.log 2024-09-18T00:42:30.481 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.57478.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99851.log 2024-09-18T00:42:30.482 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.141107.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75888.log 2024-09-18T00:42:30.483 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99851.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70241.log.gz 2024-09-18T00:42:30.483 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70510.log.gz 2024-09-18T00:42:30.483 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66018.log 2024-09-18T00:42:30.485 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75888.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143495.log 2024-09-18T00:42:30.485 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109283.log 2024-09-18T00:42:30.485 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143495.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60212.log 2024-09-18T00:42:30.485 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76380.log 2024-09-18T00:42:30.486 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.60212.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122342.log 2024-09-18T00:42:30.486 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76380.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45742.log 2024-09-18T00:42:30.487 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85285.log.gz 2024-09-18T00:42:30.487 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122342.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134096.log 2024-09-18T00:42:30.487 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63037.log 2024-09-18T00:42:30.488 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65725.log 2024-09-18T00:42:30.489 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63037.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128997.log 2024-09-18T00:42:30.490 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65725.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94172.log 2024-09-18T00:42:30.490 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128997.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.128997.log.gz 2024-09-18T00:42:30.490 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55701.log 2024-09-18T00:42:30.491 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.94172.log: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36187.log.gz 2024-09-18T00:42:30.491 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120881.log 2024-09-18T00:42:30.492 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87019.log 2024-09-18T00:42:30.492 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149783.log 2024-09-18T00:42:30.493 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113662.log 2024-09-18T00:42:30.494 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149783.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121922.log 2024-09-18T00:42:30.494 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137844.log 2024-09-18T00:42:30.495 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121922.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133227.log 2024-09-18T00:42:30.495 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42073.log 2024-09-18T00:42:30.496 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39634.log.gz 2024-09-18T00:42:30.496 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103368.log 2024-09-18T00:42:30.497 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42073.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141656.log 2024-09-18T00:42:30.497 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103368.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.103368.log.gz 2024-09-18T00:42:30.498 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141180.log 2024-09-18T00:42:30.498 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54993.log 2024-09-18T00:42:30.499 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141180.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89550.log 2024-09-18T00:42:30.500 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131660.log 2024-09-18T00:42:30.500 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65100.log 2024-09-18T00:42:30.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131660.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90012.log 2024-09-18T00:42:30.501 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37364.log 2024-09-18T00:42:30.502 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126164.log 2024-09-18T00:42:30.502 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87101.log 2024-09-18T00:42:30.503 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70697.log 2024-09-18T00:42:30.504 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135139.log 2024-09-18T00:42:30.504 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70697.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48043.log.gz 2024-09-18T00:42:30.504 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63079.log 2024-09-18T00:42:30.505 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87761.log 2024-09-18T00:42:30.505 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63079.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48589.log 2024-09-18T00:42:30.506 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87761.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97856.log 2024-09-18T00:42:30.507 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.48589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111772.log 2024-09-18T00:42:30.508 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74078.log 2024-09-18T00:42:30.508 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142799.log.gz 2024-09-18T00:42:30.508 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33851.log 2024-09-18T00:42:30.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74078.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33811.log 2024-09-18T00:42:30.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33851.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33851.log.gz 2024-09-18T00:42:30.509 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135074.log 2024-09-18T00:42:30.509 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33811.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33811.log.gz 2024-09-18T00:42:30.510 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77619.log 2024-09-18T00:42:30.511 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135074.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127220.log 2024-09-18T00:42:30.511 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77619.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121671.log 2024-09-18T00:42:30.512 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.153903.log 2024-09-18T00:42:30.513 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52551.log 2024-09-18T00:42:30.513 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.153903.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.153903.log.gz 2024-09-18T00:42:30.513 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63035.log 2024-09-18T00:42:30.513 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52551.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.66773.log.gz 2024-09-18T00:42:30.514 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.116229.log 2024-09-18T00:42:30.515 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63035.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124065.log 2024-09-18T00:42:30.516 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.116229.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73887.log 2024-09-18T00:42:30.516 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119218.log 2024-09-18T00:42:30.516 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73887.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134697.log.gz 2024-09-18T00:42:30.517 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75633.log 2024-09-18T00:42:30.517 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119218.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45580.log 2024-09-18T00:42:30.518 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152403.log 2024-09-18T00:42:30.518 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.45580.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84302.log 2024-09-18T00:42:30.519 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152403.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33850.log 2024-09-18T00:42:30.520 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126293.log.gz 2024-09-18T00:42:30.520 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84302.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43332.log 2024-09-18T00:42:30.521 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33850.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33850.log.gz 2024-09-18T00:42:30.521 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98946.log 2024-09-18T00:42:30.522 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130389.log 2024-09-18T00:42:30.522 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40131.log.gz 2024-09-18T00:42:30.522 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98946.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107464.log 2024-09-18T00:42:30.523 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63268.log 2024-09-18T00:42:30.523 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78625.log 2024-09-18T00:42:30.524 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129331.log 2024-09-18T00:42:30.524 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78625.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94541.log 2024-09-18T00:42:30.525 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.129331.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125560.log.gz 2024-09-18T00:42:30.526 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125558.log.gz 2024-09-18T00:42:30.526 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116928.log 2024-09-18T00:42:30.527 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.94541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47091.log 2024-09-18T00:42:30.527 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.116928.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122553.log 2024-09-18T00:42:30.528 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.47091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67886.log 2024-09-18T00:42:30.529 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122553.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124904.log.gz 2024-09-18T00:42:30.529 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96339.log 2024-09-18T00:42:30.529 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.67886.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.67886.log.gz 2024-09-18T00:42:30.529 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38729.log 2024-09-18T00:42:30.530 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96339.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153473.log 2024-09-18T00:42:30.530 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76380.log.gz 2024-09-18T00:42:30.531 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76148.log 2024-09-18T00:42:30.531 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.153473.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45042.log 2024-09-18T00:42:30.532 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76148.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135640.log 2024-09-18T00:42:30.533 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.45042.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69649.log 2024-09-18T00:42:30.533 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135640.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79132.log.gz 2024-09-18T00:42:30.534 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47968.log 2024-09-18T00:42:30.534 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69649.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139623.log 2024-09-18T00:42:30.535 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47968.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128135.log 2024-09-18T00:42:30.535 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139623.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105909.log 2024-09-18T00:42:30.536 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128135.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.77010.log.gz 2024-09-18T00:42:30.536 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89184.log 2024-09-18T00:42:30.537 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.105909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.139934.log 2024-09-18T00:42:30.537 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.153473.log.gz 2024-09-18T00:42:30.538 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89184.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.154487.log 2024-09-18T00:42:30.538 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.139934.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.100883.log.gz 2024-09-18T00:42:30.539 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134370.log 2024-09-18T00:42:30.539 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.154487.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.154487.log.gz 2024-09-18T00:42:30.539 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48637.log 2024-09-18T00:42:30.540 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134370.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.99020.log.gz 2024-09-18T00:42:30.540 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122175.log 2024-09-18T00:42:30.541 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41276.log 2024-09-18T00:42:30.542 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77262.log 2024-09-18T00:42:30.542 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99851.log.gz 2024-09-18T00:42:30.542 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41276.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127069.log 2024-09-18T00:42:30.543 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145357.log 2024-09-18T00:42:30.544 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.103993.log 2024-09-18T00:42:30.545 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63878.log 2024-09-18T00:42:30.545 INFO:teuthology.orchestra.run.smithi192.stderr: 57.7% -- replaced with /var/log/ceph/cluster2-client.mirror.35556.log.gz 2024-09-18T00:42:30.545 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.103993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65895.log 2024-09-18T00:42:30.546 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63878.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39360.log 2024-09-18T00:42:30.547 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37082.log 2024-09-18T00:42:30.547 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109509.log.gz 2024-09-18T00:42:30.547 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88667.log 2024-09-18T00:42:30.548 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88900.log 2024-09-18T00:42:30.548 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147014.log 2024-09-18T00:42:30.549 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88900.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56906.log 2024-09-18T00:42:30.550 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147014.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.87101.log.gz 2024-09-18T00:42:30.550 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33508.log 2024-09-18T00:42:30.550 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.79843.log.gz 2024-09-18T00:42:30.551 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.56906.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.147014.log.gz 2024-09-18T00:42:30.551 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78005.log 2024-09-18T00:42:30.551 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33508.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121671.log.gz 2024-09-18T00:42:30.552 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108704.log 2024-09-18T00:42:30.553 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78005.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65539.log 2024-09-18T00:42:30.553 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.108704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.140127.log 2024-09-18T00:42:30.554 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121922.log.gz 2024-09-18T00:42:30.554 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36575.log 2024-09-18T00:42:30.555 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.140127.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136124.log 2024-09-18T00:42:30.555 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.36575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53138.log 2024-09-18T00:42:30.556 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70396.log 2024-09-18T00:42:30.557 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113746.log 2024-09-18T00:42:30.557 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.70396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67835.log 2024-09-18T00:42:30.557 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113746.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51165.log.gz 2024-09-18T00:42:30.558 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149472.log 2024-09-18T00:42:30.558 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.67835.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.67835.log.gz 2024-09-18T00:42:30.559 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112901.log 2024-09-18T00:42:30.560 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149472.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91375.log 2024-09-18T00:42:30.560 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.112901.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72492.log 2024-09-18T00:42:30.560 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91375.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91375.log.gz 2024-09-18T00:42:30.561 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.87273.log.gz 2024-09-18T00:42:30.561 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124946.log 2024-09-18T00:42:30.562 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.72492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117813.log 2024-09-18T00:42:30.562 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.81675.log 2024-09-18T00:42:30.563 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.117813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109751.log 2024-09-18T00:42:30.564 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.81675.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.81675.log.gz 2024-09-18T00:42:30.564 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78374.log 2024-09-18T00:42:30.564 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117293.log 2024-09-18T00:42:30.565 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98046.log 2024-09-18T00:42:30.565 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.117293.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.117293.log.gz 2024-09-18T00:42:30.566 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.92618.log 2024-09-18T00:42:30.566 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98046.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113281.log.gz 2024-09-18T00:42:30.567 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119528.log 2024-09-18T00:42:30.568 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.92618.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127329.log 2024-09-18T00:42:30.569 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119528.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113957.log.gz 2024-09-18T00:42:30.569 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47941.log 2024-09-18T00:42:30.569 INFO:teuthology.orchestra.run.smithi192.stderr: 67.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129787.log.gz 2024-09-18T00:42:30.570 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127329.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51795.log 2024-09-18T00:42:30.570 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.47941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37384.log 2024-09-18T00:42:30.571 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51795.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114586.log 2024-09-18T00:42:30.571 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124508.log.gz 2024-09-18T00:42:30.571 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96907.log 2024-09-18T00:42:30.572 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114586.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114752.log 2024-09-18T00:42:30.573 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96907.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43332.log.gz 2024-09-18T00:42:30.574 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68775.log 2024-09-18T00:42:30.574 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114752.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93665.log 2024-09-18T00:42:30.575 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.68775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35454.log 2024-09-18T00:42:30.575 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93665.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.93665.log.gz 2024-09-18T00:42:30.575 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86143.log 2024-09-18T00:42:30.575 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41603.log.gz 2024-09-18T00:42:30.576 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.35454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134005.log 2024-09-18T00:42:30.577 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125620.log 2024-09-18T00:42:30.577 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134005.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.116229.log.gz 2024-09-18T00:42:30.577 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114164.log 2024-09-18T00:42:30.578 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125620.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75982.log 2024-09-18T00:42:30.579 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114164.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111438.log 2024-09-18T00:42:30.579 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75982.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120842.log.gz 2024-09-18T00:42:30.579 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.154098.log 2024-09-18T00:42:30.580 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111438.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97729.log 2024-09-18T00:42:30.581 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.154098.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.154098.log.gz 2024-09-18T00:42:30.581 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69844.log 2024-09-18T00:42:30.582 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33725.log 2024-09-18T00:42:30.583 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122594.log.gz 2024-09-18T00:42:30.583 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69844.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.69844.log.gz 2024-09-18T00:42:30.583 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69870.log 2024-09-18T00:42:30.583 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.141180.log.gz 2024-09-18T00:42:30.584 INFO:teuthology.orchestra.run.smithi192.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.105909.log.gz 2024-09-18T00:42:30.584 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33725.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33725.log.gz 2024-09-18T00:42:30.585 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97923.log 2024-09-18T00:42:30.585 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69870.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.69870.log.gz 2024-09-18T00:42:30.585 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141428.log 2024-09-18T00:42:30.586 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.97923.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133877.log 2024-09-18T00:42:30.587 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141428.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75888.log.gz 2024-09-18T00:42:30.587 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129901.log 2024-09-18T00:42:30.587 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102561.log 2024-09-18T00:42:30.588 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.81753.log.gz 2024-09-18T00:42:30.588 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.129901.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.142687.log.gz 2024-09-18T00:42:30.588 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.117226.log 2024-09-18T00:42:30.589 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143590.log 2024-09-18T00:42:30.589 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.117226.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.117226.log.gz 2024-09-18T00:42:30.589 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131417.log 2024-09-18T00:42:30.590 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143590.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.127069.log.gz 2024-09-18T00:42:30.590 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109309.log 2024-09-18T00:42:30.591 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131417.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67595.log 2024-09-18T00:42:30.592 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73886.log 2024-09-18T00:42:30.593 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67595.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82115.log 2024-09-18T00:42:30.593 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88180.log 2024-09-18T00:42:30.594 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.82115.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128583.log 2024-09-18T00:42:30.594 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88180.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.103993.log.gz 2024-09-18T00:42:30.595 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109757.log 2024-09-18T00:42:30.595 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150056.log 2024-09-18T00:42:30.596 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109757.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129974.log 2024-09-18T00:42:30.597 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147262.log 2024-09-18T00:42:30.597 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.129974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93236.log 2024-09-18T00:42:30.598 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100463.log 2024-09-18T00:42:30.599 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.93236.log: gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-18T00:42:30.599 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.147262.log.gz 2024-09-18T00:42:30.599 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100463.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58080.log 2024-09-18T00:42:30.600 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65390.log.gz 2024-09-18T00:42:30.600 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74876.log 2024-09-18T00:42:30.600 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.58080.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.58080.log.gz 2024-09-18T00:42:30.602 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59096.log 2024-09-18T00:42:30.602 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74876.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.56906.log.gz 2024-09-18T00:42:30.602 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64991.log.gz 2024-09-18T00:42:30.602 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100648.log 2024-09-18T00:42:30.602 INFO:teuthology.orchestra.run.smithi192.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70396.log.gz 2024-09-18T00:42:30.603 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.59096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77199.log 2024-09-18T00:42:30.604 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123139.log 2024-09-18T00:42:30.605 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41442.log 2024-09-18T00:42:30.605 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123139.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67491.log.gz 2024-09-18T00:42:30.605 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103724.log 2024-09-18T00:42:30.606 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41442.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140162.log 2024-09-18T00:42:30.606 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.103724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153795.log 2024-09-18T00:42:30.607 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.140162.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44497.log 2024-09-18T00:42:30.607 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.153795.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.153795.log.gz 2024-09-18T00:42:30.608 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.152582.log 2024-09-18T00:42:30.609 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.44497.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.44497.log.gz 2024-09-18T00:42:30.609 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83320.log 2024-09-18T00:42:30.609 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.152582.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63037.log.gz 2024-09-18T00:42:30.609 INFO:teuthology.orchestra.run.smithi192.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.152582.log.gz 2024-09-18T00:42:30.609 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-18T00:42:30.610 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84131.log 2024-09-18T00:42:30.610 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.83320.log: 79.8% -- replaced with /var/log/ceph/cluster1-client.mirror.131417.log.gz 2024-09-18T00:42:30.610 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131346.log.gz 2024-09-18T00:42:30.611 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70443.log 2024-09-18T00:42:30.612 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118021.log 2024-09-18T00:42:30.612 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.70443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90558.log 2024-09-18T00:42:30.616 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143077.log 2024-09-18T00:42:30.619 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125017.log 2024-09-18T00:42:30.619 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143077.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63666.log 2024-09-18T00:42:30.619 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125017.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51753.log 2024-09-18T00:42:30.619 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63666.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45748.log 2024-09-18T00:42:30.620 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96098.log 2024-09-18T00:42:30.620 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45748.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79471.log 2024-09-18T00:42:30.620 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87162.log 2024-09-18T00:42:30.620 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141040.log 2024-09-18T00:42:30.620 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.87162.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71669.log 2024-09-18T00:42:30.620 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54105.log 2024-09-18T00:42:30.622 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60296.log 2024-09-18T00:42:30.622 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.46429.log 2024-09-18T00:42:30.622 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60296.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89319.log 2024-09-18T00:42:30.622 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.46429.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.46429.log.gz 2024-09-18T00:42:30.622 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119683.log 2024-09-18T00:42:30.623 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85737.log 2024-09-18T00:42:30.623 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119683.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.128350.log 2024-09-18T00:42:30.624 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109643.log 2024-09-18T00:42:30.624 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64339.log.gz 2024-09-18T00:42:30.624 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.128350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71294.log 2024-09-18T00:42:30.625 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109643.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64381.log.gz 2024-09-18T00:42:30.625 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113241.log 2024-09-18T00:42:30.626 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.71294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109541.log 2024-09-18T00:42:30.627 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71158.log 2024-09-18T00:42:30.627 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89253.log 2024-09-18T00:42:30.628 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.71158.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74876.log.gz 2024-09-18T00:42:30.628 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74311.log 2024-09-18T00:42:30.628 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.89253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121798.log 2024-09-18T00:42:30.629 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74311.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.153927.log 2024-09-18T00:42:30.630 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.121798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52970.log 2024-09-18T00:42:30.630 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.153927.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.153927.log.gz 2024-09-18T00:42:30.630 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126844.log 2024-09-18T00:42:30.631 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110762.log 2024-09-18T00:42:30.631 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64717.log.gz 2024-09-18T00:42:30.631 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.126844.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.126844.log.gz 2024-09-18T00:42:30.632 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122973.log 2024-09-18T00:42:30.632 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110762.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141086.log 2024-09-18T00:42:30.633 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122973.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35617.log 2024-09-18T00:42:30.633 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120006.log 2024-09-18T00:42:30.634 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.35617.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.45325.log 2024-09-18T00:42:30.634 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120006.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139209.log 2024-09-18T00:42:30.636 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.45325.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92577.log 2024-09-18T00:42:30.636 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62827.log 2024-09-18T00:42:30.636 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88180.log.gz 2024-09-18T00:42:30.636 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92577.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67237.log 2024-09-18T00:42:30.637 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62827.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116557.log 2024-09-18T00:42:30.637 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67237.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118041.log 2024-09-18T00:42:30.638 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.116557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150978.log 2024-09-18T00:42:30.638 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.118041.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151846.log 2024-09-18T00:42:30.639 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34735.log 2024-09-18T00:42:30.640 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.151846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88120.log 2024-09-18T00:42:30.640 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.34735.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34735.log.gz 2024-09-18T00:42:30.640 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.admin.70443.log.gz 2024-09-18T00:42:30.640 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.154361.log 2024-09-18T00:42:30.640 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74081.log 2024-09-18T00:42:30.641 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.154361.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.154361.log.gz 2024-09-18T00:42:30.641 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134234.log.gz 2024-09-18T00:42:30.642 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94359.log 2024-09-18T00:42:30.642 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65389.log 2024-09-18T00:42:30.643 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.94359.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33483.log 2024-09-18T00:42:30.643 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.150483.log 2024-09-18T00:42:30.644 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33483.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33483.log.gz 2024-09-18T00:42:30.645 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124109.log 2024-09-18T00:42:30.645 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.150483.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.131660.log.gz 2024-09-18T00:42:30.645 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.80017.log 2024-09-18T00:42:30.646 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124109.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110594.log 2024-09-18T00:42:30.646 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.80017.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86227.log 2024-09-18T00:42:30.647 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110594.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72367.log 2024-09-18T00:42:30.648 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102687.log 2024-09-18T00:42:30.648 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.72367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86752.log 2024-09-18T00:42:30.649 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102687.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.133877.log.gz 2024-09-18T00:42:30.649 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115053.log 2024-09-18T00:42:30.649 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123982.log 2024-09-18T00:42:30.650 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.115053.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.115053.log.gz 2024-09-18T00:42:30.650 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74523.log 2024-09-18T00:42:30.651 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77475.log 2024-09-18T00:42:30.651 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124862.log.gz 2024-09-18T00:42:30.653 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74523.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100417.log 2024-09-18T00:42:30.653 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77475.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45580.log.gz 2024-09-18T00:42:30.653 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125262.log 2024-09-18T00:42:30.654 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100417.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83404.log 2024-09-18T00:42:30.654 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107699.log 2024-09-18T00:42:30.655 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.134506.log.gz 2024-09-18T00:42:30.655 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83404.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76842.log 2024-09-18T00:42:30.655 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.107699.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147474.log 2024-09-18T00:42:30.656 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134930.log 2024-09-18T00:42:30.656 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147474.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140110.log 2024-09-18T00:42:30.657 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134930.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78104.log 2024-09-18T00:42:30.657 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.147474.log.gz 2024-09-18T00:42:30.658 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.140110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48330.log 2024-09-18T00:42:30.659 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78104.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65895.log.gz 2024-09-18T00:42:30.659 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73283.log 2024-09-18T00:42:30.660 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74792.log 2024-09-18T00:42:30.660 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63458.log 2024-09-18T00:42:30.661 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137468.log 2024-09-18T00:42:30.661 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40855.log 90.0% 2024-09-18T00:42:30.661 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.145145.log.gz 2024-09-18T00:42:30.662 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137468.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98678.log 2024-09-18T00:42:30.663 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125283.log 2024-09-18T00:42:30.663 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.45042.log.gz 2024-09-18T00:42:30.664 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98678.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53876.log 2024-09-18T00:42:30.664 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.89382.log.gz 2024-09-18T00:42:30.664 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76694.log 2024-09-18T00:42:30.665 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102012.log 2024-09-18T00:42:30.666 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76694.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84167.log 2024-09-18T00:42:30.667 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54735.log 2024-09-18T00:42:30.668 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84167.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99098.log 2024-09-18T00:42:30.668 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117384.log 2024-09-18T00:42:30.669 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99098.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63374.log 2024-09-18T00:42:30.669 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.117384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102642.log 2024-09-18T00:42:30.670 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63374.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152995.log 2024-09-18T00:42:30.670 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102642.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47601.log 2024-09-18T00:42:30.671 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.152995.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100463.log.gz 2024-09-18T00:42:30.671 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52889.log 2024-09-18T00:42:30.671 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.152995.log.gz 2024-09-18T00:42:30.672 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.47601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78671.log 2024-09-18T00:42:30.673 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52889.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83708.log 2024-09-18T00:42:30.674 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78625.log.gz 2024-09-18T00:42:30.674 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78671.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.admin.35454.log.gz 2024-09-18T00:42:30.674 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137385.log 2024-09-18T00:42:30.675 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83708.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79214.log 2024-09-18T00:42:30.675 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137385.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78005.log.gz 2024-09-18T00:42:30.676 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61920.log 2024-09-18T00:42:30.676 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40641.log 2024-09-18T00:42:30.677 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.61920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95383.log 2024-09-18T00:42:30.678 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111225.log 2024-09-18T00:42:30.678 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145856.log 2024-09-18T00:42:30.679 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.33559.log 2024-09-18T00:42:30.680 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100074.log 2024-09-18T00:42:30.680 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.33559.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.33559.log.gz 2024-09-18T00:42:30.680 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46776.log 2024-09-18T00:42:30.681 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100074.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43123.log 2024-09-18T00:42:30.681 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.46776.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128579.log 2024-09-18T00:42:30.682 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43123.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124718.log 2024-09-18T00:42:30.683 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.128579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.80792.log 2024-09-18T00:42:30.683 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124718.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69149.log 2024-09-18T00:42:30.684 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.80792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134907.log 2024-09-18T00:42:30.685 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69149.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111982.log 2024-09-18T00:42:30.686 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37034.log 2024-09-18T00:42:30.686 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111982.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114586.log.gz 2024-09-18T00:42:30.686 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152922.log 2024-09-18T00:42:30.686 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.37034.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.139623.log.gz 2024-09-18T00:42:30.686 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147353.log 2024-09-18T00:42:30.687 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.152922.log: 66.3% -- replaced with /var/log/ceph/cluster1-client.mirror.152922.log.gz 2024-09-18T00:42:30.687 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76340.log 2024-09-18T00:42:30.688 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63751.log 2024-09-18T00:42:30.689 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74162.log 2024-09-18T00:42:30.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82763.log 2024-09-18T00:42:30.690 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74162.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51330.log 2024-09-18T00:42:30.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.82763.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112901.log.gz 2024-09-18T00:42:30.691 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59355.log 2024-09-18T00:42:30.691 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79156.log 2024-09-18T00:42:30.692 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.59355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111018.log 2024-09-18T00:42:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78945.log 2024-09-18T00:42:30.693 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123982.log.gz 2024-09-18T00:42:30.694 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82069.log 2024-09-18T00:42:30.694 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.78945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35915.log 2024-09-18T00:42:30.695 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.82069.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75298.log 2024-09-18T00:42:30.695 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.35915.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135390.log 2024-09-18T00:42:30.696 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75298.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62197.log.gz 2024-09-18T00:42:30.696 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35857.log 2024-09-18T00:42:30.696 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145063.log 2024-09-18T00:42:30.697 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.35857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150598.log 2024-09-18T00:42:30.697 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.140110.log.gz 2024-09-18T00:42:30.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145063.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109543.log 2024-09-18T00:42:30.698 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150598.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.150598.log.gz 2024-09-18T00:42:30.698 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79458.log 2024-09-18T00:42:30.699 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109543.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136847.log 2024-09-18T00:42:30.699 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79458.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.79458.log.gz 2024-09-18T00:42:30.699 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66421.log 2024-09-18T00:42:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136847.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134370.log.gz 2024-09-18T00:42:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94795.log 2024-09-18T00:42:30.701 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.66421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.57032.log 2024-09-18T00:42:30.702 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.94795.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44511.log 2024-09-18T00:42:30.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.57032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67028.log 2024-09-18T00:42:30.703 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64675.log 2024-09-18T00:42:30.704 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149365.log 2024-09-18T00:42:30.704 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64675.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52340.log.gz 2024-09-18T00:42:30.704 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135684.log 2024-09-18T00:42:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.95597.log 2024-09-18T00:42:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135684.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35191.log 2024-09-18T00:42:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.95597.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131127.log 2024-09-18T00:42:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.35191.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35191.log.gz 2024-09-18T00:42:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.94473.log 2024-09-18T00:42:30.707 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52889.log.gz 2024-09-18T00:42:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131127.log: gzip 87.4% -5 -- replaced with /var/log/ceph/cluster1-client.mirror.96098.log.gz --verbose 90.0% 2024-09-18T00:42:30.708 INFO:teuthology.orchestra.run.smithi192.stderr: -- -- replaced with /var/log/ceph/cluster1-client.mirror.52970.log.gz /var/log/ceph/cluster1-client.mirror.54188.log 2024-09-18T00:42:30.708 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:30.709 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.94473.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91320.log 2024-09-18T00:42:30.709 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75318.log 2024-09-18T00:42:30.710 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72015.log 2024-09-18T00:42:30.711 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141636.log 2024-09-18T00:42:30.711 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95447.log.gz 2024-09-18T00:42:30.711 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.72015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113492.log 2024-09-18T00:42:30.712 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141636.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.93018.log 2024-09-18T00:42:30.713 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56634.log 2024-09-18T00:42:30.713 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55743.log.gz 2024-09-18T00:42:30.714 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.93018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114121.log 2024-09-18T00:42:30.714 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56634.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104375.log 2024-09-18T00:42:30.715 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43476.log 2024-09-18T00:42:30.715 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.104375.log: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55808.log.gz 2024-09-18T00:42:30.716 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.56688.log 2024-09-18T00:42:30.716 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55280.log.gz 2024-09-18T00:42:30.716 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53516.log 2024-09-18T00:42:30.717 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.56688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.58054.log 2024-09-18T00:42:30.718 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53516.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136439.log.gz 2024-09-18T00:42:30.718 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137659.log 2024-09-18T00:42:30.718 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.58054.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.58054.log.gz 2024-09-18T00:42:30.719 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111880.log 2024-09-18T00:42:30.720 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79724.log 2024-09-18T00:42:30.720 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111880.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.83320.log.gz 2024-09-18T00:42:30.721 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131162.log 2024-09-18T00:42:30.721 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.83429.log.gz 2024-09-18T00:42:30.721 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.79724.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.83602.log.gz 2024-09-18T00:42:30.721 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.83556.log.gz 2024-09-18T00:42:30.722 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50450.log 2024-09-18T00:42:30.722 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.83624.log.gz 2024-09-18T00:42:30.723 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131162.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.131089.log.gz 2024-09-18T00:42:30.723 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82498.log 2024-09-18T00:42:30.723 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50450.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98568.log 2024-09-18T00:42:30.724 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.82498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135872.log 2024-09-18T00:42:30.725 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98568.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130521.log 2024-09-18T00:42:30.725 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134005.log.gz 2024-09-18T00:42:30.725 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135872.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131264.log 2024-09-18T00:42:30.726 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130521.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47621.log 2024-09-18T00:42:30.726 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.131264.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110807.log 2024-09-18T00:42:30.727 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.47621.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133960.log 2024-09-18T00:42:30.728 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110807.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77682.log 2024-09-18T00:42:30.729 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.133960.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147361.log 2024-09-18T00:42:30.729 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59313.log.gz 2024-09-18T00:42:30.729 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59469.log.gz 2024-09-18T00:42:30.729 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77682.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153836.log 2024-09-18T00:42:30.730 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84633.log 2024-09-18T00:42:30.731 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.153836.log: 66.3% -- replaced with /var/log/ceph/cluster1-client.mirror.153836.log.gz 2024-09-18T00:42:30.731 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67449.log 2024-09-18T00:42:30.732 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84633.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98108.log 2024-09-18T00:42:30.733 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.67889.log 2024-09-18T00:42:30.733 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98108.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.84131.log.gz 2024-09-18T00:42:30.733 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77263.log 2024-09-18T00:42:30.734 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131162.log.gz 2024-09-18T00:42:30.734 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.84167.log.gz 2024-09-18T00:42:30.734 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.67889.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.67889.log.gz 2024-09-18T00:42:30.735 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62575.log 2024-09-18T00:42:30.735 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.77263.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145692.log 2024-09-18T00:42:30.736 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100752.log 2024-09-18T00:42:30.737 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83735.log 2024-09-18T00:42:30.737 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142431.log 2024-09-18T00:42:30.738 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54485.log 2024-09-18T00:42:30.738 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74228.log 2024-09-18T00:42:30.739 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.54485.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76990.log 2024-09-18T00:42:30.739 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134610.log 2024-09-18T00:42:30.740 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76990.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118800.log 2024-09-18T00:42:30.741 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.134610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83976.log 2024-09-18T00:42:30.741 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.118800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38135.log 2024-09-18T00:42:30.742 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102772.log 2024-09-18T00:42:30.743 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100250.log 2024-09-18T00:42:30.743 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61536.log.gz 2024-09-18T00:42:30.743 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141864.log 2024-09-18T00:42:30.743 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100250.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35266.log 2024-09-18T00:42:30.744 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141864.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46614.log 2024-09-18T00:42:30.745 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.35266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123057.log 2024-09-18T00:42:30.745 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.46614.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74229.log 2024-09-18T00:42:30.745 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127515.log 2024-09-18T00:42:30.746 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74229.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96339.log.gz 2024-09-18T00:42:30.746 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74270.log 2024-09-18T00:42:30.747 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127515.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95640.log 2024-09-18T00:42:30.748 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142733.log 2024-09-18T00:42:30.750 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.95640.log: gzip -5 --verbose -- 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97729.log.gz 2024-09-18T00:42:30.751 INFO:teuthology.orchestra.run.smithi192.stderr: /var/log/ceph/cluster1-client.mirror.73664.log 2024-09-18T00:42:30.751 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.142733.log: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.68443.log.gz 2024-09-18T00:42:30.751 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114583.log 2024-09-18T00:42:30.751 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.73664.log: gzip -5 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.68460.log.gz 2024-09-18T00:42:30.751 INFO:teuthology.orchestra.run.smithi192.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.143454.log 2024-09-18T00:42:30.752 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.114583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138807.log 2024-09-18T00:42:30.752 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.143454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53099.log 2024-09-18T00:42:30.753 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138807.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116820.log 2024-09-18T00:42:30.753 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53099.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99635.log 2024-09-18T00:42:30.754 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.116820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145155.log 2024-09-18T00:42:30.754 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99635.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.69389.log.gz 2024-09-18T00:42:30.754 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141127.log 2024-09-18T00:42:30.755 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145155.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.69651.log.gz 2024-09-18T00:42:30.755 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43163.log 2024-09-18T00:42:30.755 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.141127.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.69948.log 2024-09-18T00:42:30.756 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119305.log 2024-09-18T00:42:30.756 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.69948.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.69948.log.gz 2024-09-18T00:42:30.757 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68628.log 2024-09-18T00:42:30.757 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.119305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55986.log 2024-09-18T00:42:30.758 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68628.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109351.log 2024-09-18T00:42:30.759 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.55986.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.55986.log.gz 2024-09-18T00:42:30.759 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102620.log 2024-09-18T00:42:30.760 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.109351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84234.log 2024-09-18T00:42:30.760 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102620.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46547.log 2024-09-18T00:42:30.761 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84234.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63498.log 2024-09-18T00:42:30.762 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.46547.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105159.log 2024-09-18T00:42:30.762 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43474.log 2024-09-18T00:42:30.763 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.105159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.105795.log 2024-09-18T00:42:30.764 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43474.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122218.log.gz 2024-09-18T00:42:30.764 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129465.log 2024-09-18T00:42:30.765 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.105795.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55281.log 2024-09-18T00:42:30.765 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.129465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75763.log 2024-09-18T00:42:30.765 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55281.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58526.log 2024-09-18T00:42:30.766 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137870.log 2024-09-18T00:42:30.766 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.58526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63793.log 2024-09-18T00:42:30.767 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48045.log 2024-09-18T00:42:30.768 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63793.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86227.log.gz 2024-09-18T00:42:30.768 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.82673.log 2024-09-18T00:42:30.769 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88313.log 2024-09-18T00:42:30.769 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.82673.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116883.log 2024-09-18T00:42:30.770 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.88313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148146.log 2024-09-18T00:42:30.771 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.116883.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57550.log 2024-09-18T00:42:30.771 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134907.log.gz 2024-09-18T00:42:30.771 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.150778.log 2024-09-18T00:42:30.772 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.57550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149534.log 2024-09-18T00:42:30.773 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.150778.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50516.log 2024-09-18T00:42:30.774 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.151090.log 2024-09-18T00:42:30.774 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106540.log 2024-09-18T00:42:30.775 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.151090.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74270.log.gz 2024-09-18T00:42:30.776 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.68480.log 2024-09-18T00:42:30.776 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.106540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99799.log 2024-09-18T00:42:30.776 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.68480.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.74792.log.gz 2024-09-18T00:42:30.777 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.68710.log 2024-09-18T00:42:30.777 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150547.log 2024-09-18T00:42:30.778 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.68710.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137870.log.gz 2024-09-18T00:42:30.779 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148376.log 2024-09-18T00:42:30.779 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150547.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.150547.log.gz 2024-09-18T00:42:30.779 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65348.log 2024-09-18T00:42:30.779 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148376.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.147353.log.gz 2024-09-18T00:42:30.780 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102768.log 2024-09-18T00:42:30.781 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46593.log 2024-09-18T00:42:30.782 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102768.log: 87.4%gzip -- replaced with /var/log/ceph/cluster2-client.admin.128350.log.gz 2024-09-18T00:42:30.782 INFO:teuthology.orchestra.run.smithi192.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99199.log 2024-09-18T00:42:30.782 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.46593.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53415.log 2024-09-18T00:42:30.783 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46520.log 2024-09-18T00:42:30.783 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53415.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128561.log 2024-09-18T00:42:30.784 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.46520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138162.log 2024-09-18T00:42:30.785 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.128561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53412.log 2024-09-18T00:42:30.785 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67028.log.gz 2024-09-18T00:42:30.786 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.138162.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74698.log 2024-09-18T00:42:30.787 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86730.log 2024-09-18T00:42:30.787 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74698.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91320.log.gz 2024-09-18T00:42:30.787 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91307.log 2024-09-18T00:42:30.788 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86730.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.91950.log 2024-09-18T00:42:30.789 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91307.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.91307.log.gz 2024-09-18T00:42:30.789 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145733.log 2024-09-18T00:42:30.790 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.91950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70355.log 2024-09-18T00:42:30.791 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145733.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147558.log 2024-09-18T00:42:30.791 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78671.log.gz 2024-09-18T00:42:30.791 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.70355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52341.log 2024-09-18T00:42:30.792 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137237.log 2024-09-18T00:42:30.792 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39861.log 2024-09-18T00:42:30.793 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137237.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47347.log 2024-09-18T00:42:30.794 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39861.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.77682.log.gz 2024-09-18T00:42:30.794 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125955.log 2024-09-18T00:42:30.794 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.47347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38179.log 2024-09-18T00:42:30.795 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118555.log 2024-09-18T00:42:30.796 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47941.log.gz 2024-09-18T00:42:30.796 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.38179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74312.log 2024-09-18T00:42:30.797 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.118555.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.93099.log 2024-09-18T00:42:30.797 INFO:teuthology.orchestra.run.smithi192.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48637.log.gz 2024-09-18T00:42:30.798 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.74312.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.48589.log.gz 2024-09-18T00:42:30.798 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147583.log 2024-09-18T00:42:30.799 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.93099.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135469.log 2024-09-18T00:42:30.799 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48242.log 2024-09-18T00:42:30.800 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135469.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141200.log 2024-09-18T00:42:30.800 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.147583.log.gz 2024-09-18T00:42:30.801 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.48242.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95336.log 2024-09-18T00:42:30.802 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.141200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96270.log 2024-09-18T00:42:30.802 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.95336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65832.log 2024-09-18T00:42:30.803 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.96270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76295.log 2024-09-18T00:42:30.804 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79997.log 2024-09-18T00:42:30.804 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.136190.log.gz 2024-09-18T00:42:30.804 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76295.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124039.log 2024-09-18T00:42:30.805 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.151896.log 2024-09-18T00:42:30.805 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124039.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62324.log 2024-09-18T00:42:30.806 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.151896.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.92270.log 2024-09-18T00:42:30.806 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.62324.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98568.log.gz 2024-09-18T00:42:30.807 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67069.log 2024-09-18T00:42:30.807 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.92270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41484.log 2024-09-18T00:42:30.808 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.67069.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.98678.log.gz 2024-09-18T00:42:30.808 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145774.log 2024-09-18T00:42:30.809 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41484.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111601.log 2024-09-18T00:42:30.810 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86455.log 2024-09-18T00:42:30.810 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76424.log 2024-09-18T00:42:30.811 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86455.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71210.log 2024-09-18T00:42:30.811 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127129.log 2024-09-18T00:42:30.812 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.71210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139030.log 2024-09-18T00:42:30.813 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.127129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140838.log 2024-09-18T00:42:30.813 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.139030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148170.log 2024-09-18T00:42:30.814 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.77473.log.gz 2024-09-18T00:42:30.814 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.140838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145366.log 2024-09-18T00:42:30.815 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.148170.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115814.log 2024-09-18T00:42:30.815 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.145366.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148170.log.gz 2024-09-18T00:42:30.816 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50618.log 2024-09-18T00:42:30.816 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.115814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43846.log 2024-09-18T00:42:30.816 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.151896.log.gz 2024-09-18T00:42:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.50618.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99536.log 2024-09-18T00:42:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43846.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112442.log.gz 2024-09-18T00:42:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99073.log 2024-09-18T00:42:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99536.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125283.log.gz 2024-09-18T00:42:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119717.log 2024-09-18T00:42:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41063.log 2024-09-18T00:42:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.119717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55516.log 2024-09-18T00:42:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41063.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131043.log 2024-09-18T00:42:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.55516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94836.log 2024-09-18T00:42:30.823 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.131043.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.136847.log.gz 2024-09-18T00:42:30.824 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99492.log 2024-09-18T00:42:30.824 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.94836.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39401.log 2024-09-18T00:42:30.825 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123899.log 2024-09-18T00:42:30.825 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39401.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64298.log 2024-09-18T00:42:30.825 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.123899.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133960.log.gz 2024-09-18T00:42:30.825 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92576.log 2024-09-18T00:42:30.826 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100333.log 2024-09-18T00:42:30.826 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92576.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137385.log.gz 2024-09-18T00:42:30.826 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95967.log 2024-09-18T00:42:30.827 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.100333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110341.log 2024-09-18T00:42:30.828 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.95967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40431.log 2024-09-18T00:42:30.829 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110341.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.74081.log.gz 2024-09-18T00:42:30.829 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53727.log 2024-09-18T00:42:30.830 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63709.log 2024-09-18T00:42:30.830 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125449.log 2024-09-18T00:42:30.831 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.63709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111183.log 2024-09-18T00:42:30.832 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64908.log 2024-09-18T00:42:30.832 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.35857.log.gz 2024-09-18T00:42:30.832 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37324.log 2024-09-18T00:42:30.833 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.64908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98606.log 2024-09-18T00:42:30.833 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.37324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99470.log 2024-09-18T00:42:30.834 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.98606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42282.log 2024-09-18T00:42:30.835 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99470.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65034.log 2024-09-18T00:42:30.836 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.42282.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132476.log 2024-09-18T00:42:30.836 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99418.log 2024-09-18T00:42:30.837 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.132476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102141.log 2024-09-18T00:42:30.838 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.99418.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147012.log 2024-09-18T00:42:30.838 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102141.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41908.log 2024-09-18T00:42:30.839 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147895.log 2024-09-18T00:42:30.839 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102768.log.gz 2024-09-18T00:42:30.840 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.41908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65287.log 2024-09-18T00:42:30.840 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.147012.log.gz 2024-09-18T00:42:30.841 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124382.log 2024-09-18T00:42:30.841 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65287.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100648.log.gz 2024-09-18T00:42:30.842 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125139.log 2024-09-18T00:42:30.842 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.124382.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.147895.log.gz 2024-09-18T00:42:30.842 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130523.log 2024-09-18T00:42:30.843 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90611.log 2024-09-18T00:42:30.844 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.130523.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.149216.log 2024-09-18T00:42:30.845 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.90611.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109351.log.gz 2024-09-18T00:42:30.845 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.79568.log 2024-09-18T00:42:30.845 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.149216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52426.log 2024-09-18T00:42:30.846 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.79568.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.79568.log.gz 2024-09-18T00:42:30.846 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107310.log.gz 2024-09-18T00:42:30.846 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36990.log 2024-09-18T00:42:30.847 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.52426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144910.log 2024-09-18T00:42:30.848 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.36990.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126628.log 2024-09-18T00:42:30.848 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.144910.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116677.log 2024-09-18T00:42:30.849 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.126628.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111642.log 2024-09-18T00:42:30.850 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.56634.log.gz 2024-09-18T00:42:30.850 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53288.log 2024-09-18T00:42:30.850 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.111642.log: /var/log/ceph/cluster2-client.admin.116677.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101173.log 2024-09-18T00:42:30.851 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65204.log 2024-09-18T00:42:30.852 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101173.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80447.log 2024-09-18T00:42:30.852 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.65204.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40279.log 2024-09-18T00:42:30.853 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.80447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76104.log 2024-09-18T00:42:30.853 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.40279.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127021.log.gz 2024-09-18T00:42:30.854 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102309.log 2024-09-18T00:42:30.854 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.76104.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86835.log 2024-09-18T00:42:30.855 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122974.log 2024-09-18T00:42:30.856 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.86835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107397.log 2024-09-18T00:42:30.856 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122974.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.127129.log.gz 2024-09-18T00:42:30.856 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39799.log 2024-09-18T00:42:30.856 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64299.log.gz 2024-09-18T00:42:30.857 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.107397.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.150506.log 2024-09-18T00:42:30.858 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.39799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137952.log 2024-09-18T00:42:30.858 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.150506.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.150506.log.gz 2024-09-18T00:42:30.859 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135265.log 2024-09-18T00:42:30.860 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.137952.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102978.log 2024-09-18T00:42:30.860 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.135265.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147584.log 2024-09-18T00:42:30.860 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.102978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46661.log 2024-09-18T00:42:30.861 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.147584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.154182.log 2024-09-18T00:42:30.862 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.46661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132415.log 2024-09-18T00:42:30.862 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.77054.log.gz 2024-09-18T00:42:30.862 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.154182.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.154182.log.gz 2024-09-18T00:42:30.862 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44554.log 2024-09-18T00:42:30.863 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.132415.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104641.log 2024-09-18T00:42:30.864 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.44554.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84412.log 2024-09-18T00:42:30.864 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.88667.log.gz 2024-09-18T00:42:30.864 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.104641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75044.log 2024-09-18T00:42:30.865 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.84454.log 2024-09-18T00:42:30.865 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.75044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125180.log 2024-09-18T00:42:30.866 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.84454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84256.log 2024-09-18T00:42:30.867 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.125180.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113118.log 2024-09-18T00:42:30.867 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.84256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122410.log 2024-09-18T00:42:30.868 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.113118.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49232.log.gz 2024-09-18T00:42:30.868 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91565.log 2024-09-18T00:42:30.869 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.122410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60413.log 2024-09-18T00:42:30.870 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.91565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136775.log 2024-09-18T00:42:30.870 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.60413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101888.log 2024-09-18T00:42:30.871 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.136775.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99635.log.gz 2024-09-18T00:42:30.871 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.83271.log 2024-09-18T00:42:30.872 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.101888.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71426.log 2024-09-18T00:42:30.872 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.83271.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109541.log.gz 2024-09-18T00:42:30.872 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142251.log 2024-09-18T00:42:30.873 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.71426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53919.log 2024-09-18T00:42:30.874 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.admin.142251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51289.log 2024-09-18T00:42:30.875 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.53919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43164.log 2024-09-18T00:42:30.875 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.51289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85327.log 2024-09-18T00:42:30.876 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.43164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.92689.log 2024-09-18T00:42:30.877 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.85327.log: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.79156.log.gz 2024-09-18T00:42:30.877 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110761.log 2024-09-18T00:42:30.878 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.admin.92689.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.81199.log 2024-09-18T00:42:30.879 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.110761.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120390.log 2024-09-18T00:42:30.879 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster2-client.mirror.81199.log: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.82069.log.gz 2024-09-18T00:42:30.879 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.68710.log.gz 2024-09-18T00:42:30.881 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cluster1-client.mirror.120390.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59991.log.gz 2024-09-18T00:42:30.882 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60413.log.gz 2024-09-18T00:42:30.882 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.84256.log.gz 2024-09-18T00:42:30.884 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141636.log.gz 2024-09-18T00:42:30.886 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.96270.log.gz 2024-09-18T00:42:30.890 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97923.log.gz 2024-09-18T00:42:30.892 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53516.log.gz 2024-09-18T00:42:30.894 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79724.log.gz 2024-09-18T00:42:30.896 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44173.log.gz 2024-09-18T00:42:30.899 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.43846.log.gz 2024-09-18T00:42:30.899 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39401.log.gz 2024-09-18T00:42:30.901 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111642.log.gz 2024-09-18T00:42:30.903 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60564.log.gz 2024-09-18T00:42:30.905 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60764.log.gz 2024-09-18T00:42:30.907 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.95640.log.gz 2024-09-18T00:42:30.908 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.95967.log.gz 2024-09-18T00:42:30.912 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52341.log.gz 2024-09-18T00:42:30.916 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111880.log.gz 2024-09-18T00:42:30.922 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98046.log.gz 2024-09-18T00:42:30.922 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98108.log.gz 2024-09-18T00:42:30.924 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.141127.log.gz 2024-09-18T00:42:30.932 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123057.log.gz 2024-09-18T00:42:30.937 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.81867.log.gz 2024-09-18T00:42:30.942 INFO:teuthology.orchestra.run.smithi192.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.82498.log.gz 2024-09-18T00:42:30.942 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.admin.82525.log.gz 2024-09-18T00:42:30.946 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.105159.log.gz 2024-09-18T00:42:30.947 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.106540.log.gz 2024-09-18T00:42:30.955 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40813.log.gz 2024-09-18T00:42:30.957 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74698.log.gz 2024-09-18T00:42:30.961 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.134360.log.gz 2024-09-18T00:42:30.963 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126419.log.gz 2024-09-18T00:42:30.965 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51289.log.gz 2024-09-18T00:42:30.969 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51103.log.gz 2024-09-18T00:42:30.975 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.132415.log.gz 2024-09-18T00:42:30.984 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.87660.log.gz 2024-09-18T00:42:30.987 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102223.log.gz 2024-09-18T00:42:30.989 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87554.log.gz 2024-09-18T00:42:30.992 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.99536.log.gz 2024-09-18T00:42:30.996 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74161.log.gz 2024-09-18T00:42:31.000 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.142949.log.gz 2024-09-18T00:42:31.004 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104376.log.gz 2024-09-18T00:42:31.004 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.80447.log.gz 2024-09-18T00:42:31.007 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53728.log.gz 2024-09-18T00:42:31.008 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53012.log.gz 2024-09-18T00:42:31.009 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53057.log.gz 2024-09-18T00:42:31.012 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.83735.log.gz 2024-09-18T00:42:31.014 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.93473.log.gz 2024-09-18T00:42:31.016 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114168.log.gz 2024-09-18T00:42:31.018 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.141811.log.gz 2024-09-18T00:42:31.021 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.129290.log.gz 2024-09-18T00:42:31.021 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.129351.log.gz 2024-09-18T00:42:31.021 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.129559.log.gz 2024-09-18T00:42:31.024 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.140314.log.gz 2024-09-18T00:42:31.026 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41908.log.gz 2024-09-18T00:42:31.028 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.115600.log.gz 2024-09-18T00:42:31.030 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54188.log.gz 2024-09-18T00:42:31.032 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103379.log.gz 2024-09-18T00:42:31.033 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.81199.log.gz 2024-09-18T00:42:31.035 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.96846.log.gz 2024-09-18T00:42:31.038 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54187.log.gz 2024-09-18T00:42:31.041 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42282.log.gz 2024-09-18T00:42:31.044 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.128371.log.gz 2024-09-18T00:42:31.045 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43586.log.gz 2024-09-18T00:42:31.047 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122974.log.gz 2024-09-18T00:42:31.053 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131262.log.gz 2024-09-18T00:42:31.056 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111853.log.gz 2024-09-18T00:42:31.058 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.60518.log.gz 2024-09-18T00:42:31.060 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71908.log.gz 2024-09-18T00:42:31.062 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.119683.log.gz 2024-09-18T00:42:31.064 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120006.log.gz 2024-09-18T00:42:31.075 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.107397.log.gz 2024-09-18T00:42:31.080 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101888.log.gz 2024-09-18T00:42:31.082 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101912.log.gz 2024-09-18T00:42:31.084 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45948.log.gz 2024-09-18T00:42:31.087 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.72099.log.gz 2024-09-18T00:42:31.089 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125449.log.gz 2024-09-18T00:42:31.090 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.81955.log.gz 2024-09-18T00:42:31.096 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87397.log.gz 2024-09-18T00:42:31.105 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88311.log.gz 2024-09-18T00:42:31.106 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.92270.log.gz 2024-09-18T00:42:31.108 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76905.log.gz 2024-09-18T00:42:31.112 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.137509.log.gz 2024-09-18T00:42:31.113 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145733.log.gz 2024-09-18T00:42:31.117 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67321.log.gz 2024-09-18T00:42:31.125 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74314.log.gz 2024-09-18T00:42:31.127 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.141697.log.gz 2024-09-18T00:42:31.127 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137867.log.gz 2024-09-18T00:42:31.129 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46802.log.gz 2024-09-18T00:42:31.130 INFO:teuthology.orchestra.run.smithi096.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-18T00:42:31.132 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101511.log.gz 2024-09-18T00:42:31.135 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98820.log.gz 2024-09-18T00:42:31.138 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42283.log.gz 2024-09-18T00:42:31.139 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.116101.log.gz 2024-09-18T00:42:31.140 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.89589.log.gz 2024-09-18T00:42:31.141 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.116225.log.gz 2024-09-18T00:42:31.142 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100653.log.gz 2024-09-18T00:42:31.143 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.56818.log.gz 2024-09-18T00:42:31.147 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138013.log.gz 2024-09-18T00:42:31.151 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36248.log.gz 2024-09-18T00:42:31.153 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.39214.log.gz 2024-09-18T00:42:31.154 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71820.log.gz 2024-09-18T00:42:31.162 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78524.log.gz 2024-09-18T00:42:31.165 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102770.log.gz 2024-09-18T00:42:31.168 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109326.log.gz 2024-09-18T00:42:31.172 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90223.log.gz 2024-09-18T00:42:31.174 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102013.log.gz 2024-09-18T00:42:31.179 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41694.log.gz 2024-09-18T00:42:31.182 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65286.log.gz 2024-09-18T00:42:31.184 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124381.log.gz 2024-09-18T00:42:31.189 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50618.log.gz 2024-09-18T00:42:31.191 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50745.log.gz 2024-09-18T00:42:31.194 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126970.log.gz 2024-09-18T00:42:31.197 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.66227.log.gz 2024-09-18T00:42:31.198 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.66167.log.gz 2024-09-18T00:42:31.200 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.133668.log.gz 2024-09-18T00:42:31.204 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.139706.log.gz 2024-09-18T00:42:31.206 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.142581.log.gz 2024-09-18T00:42:31.207 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143711.log.gz 2024-09-18T00:42:31.210 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.136708.log.gz 2024-09-18T00:42:31.213 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137574.log.gz 2024-09-18T00:42:31.216 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109672.log.gz 2024-09-18T00:42:31.219 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76195.log.gz 2024-09-18T00:42:31.222 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125071.log.gz 2024-09-18T00:42:31.228 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.88499.log.gz 2024-09-18T00:42:31.231 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125492.log.gz 2024-09-18T00:42:31.234 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.69093.log.gz 2024-09-18T00:42:31.237 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126374.log.gz 2024-09-18T00:42:31.238 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.134569.log.gz 2024-09-18T00:42:31.242 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.151333.log.gz 2024-09-18T00:42:31.242 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.128469.log.gz 2024-09-18T00:42:31.246 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37145.log.gz 2024-09-18T00:42:31.249 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78248.log.gz 2024-09-18T00:42:31.251 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100796.log.gz 2024-09-18T00:42:31.254 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94473.log.gz 2024-09-18T00:42:31.256 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108743.log.gz 2024-09-18T00:42:31.260 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.49212.log.gz 2024-09-18T00:42:31.263 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.83976.log.gz 2024-09-18T00:42:31.268 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.admin.82183.log.gz 2024-09-18T00:42:31.272 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.80976.log.gz 2024-09-18T00:42:31.272 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134987.log.gz 2024-09-18T00:42:31.275 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.152351.log.gz 2024-09-18T00:42:31.278 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90973.log.gz 2024-09-18T00:42:31.280 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100547.log.gz 2024-09-18T00:42:31.282 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90768.log.gz 2024-09-18T00:42:31.282 INFO:teuthology.orchestra.run.smithi192.stderr: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95724.log.gz 2024-09-18T00:42:31.286 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.85893.log.gz 2024-09-18T00:42:31.289 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65536.log.gz 2024-09-18T00:42:31.293 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60736.log.gz 2024-09-18T00:42:31.293 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63500.log.gz 2024-09-18T00:42:31.297 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74312.log.gz 2024-09-18T00:42:31.297 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74523.log.gz 2024-09-18T00:42:31.298 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.93883.log.gz 2024-09-18T00:42:31.298 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.119307.log.gz 2024-09-18T00:42:31.302 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.38449.log.gz 2024-09-18T00:42:31.307 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65287.log.gz 2024-09-18T00:42:31.308 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64883.log.gz 2024-09-18T00:42:31.311 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40221.log.gz 2024-09-18T00:42:31.316 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76466.log.gz 2024-09-18T00:42:31.316 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76526.log.gz 2024-09-18T00:42:31.320 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62826.log.gz 2024-09-18T00:42:31.323 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62954.log.gz 2024-09-18T00:42:31.328 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40643.log.gz 2024-09-18T00:42:31.333 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66607.log.gz 2024-09-18T00:42:31.335 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.137031.log.gz 2024-09-18T00:42:31.337 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.146111.log.gz 2024-09-18T00:42:31.340 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45571.log.gz 2024-09-18T00:42:31.341 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.46007.log.gz 2024-09-18T00:42:31.342 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.134717.log.gz 2024-09-18T00:42:31.344 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136315.log.gz 2024-09-18T00:42:31.346 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125998.log.gz 2024-09-18T00:42:31.348 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51250.log.gz 2024-09-18T00:42:31.350 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53961.log.gz 2024-09-18T00:42:31.353 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138287.log.gz 2024-09-18T00:42:31.356 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86562.log.gz 2024-09-18T00:42:31.360 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.100376.log.gz 2024-09-18T00:42:31.368 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102098.log.gz 2024-09-18T00:42:31.370 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.151089.log.gz 2024-09-18T00:42:31.371 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50954.log.gz 2024-09-18T00:42:31.373 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86100.log.gz 2024-09-18T00:42:31.375 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86185.log.gz 2024-09-18T00:42:31.377 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79237.log.gz 2024-09-18T00:42:31.381 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41813.log.gz 2024-09-18T00:42:31.383 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54991.log.gz 2024-09-18T00:42:31.384 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.91923.log.gz 2024-09-18T00:42:31.384 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.91878.log.gz 2024-09-18T00:42:31.386 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94453.log.gz 2024-09-18T00:42:31.388 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.141335.log.gz 2024-09-18T00:42:31.388 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93113.log.gz 2024-09-18T00:42:31.393 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121245.log.gz 2024-09-18T00:42:31.394 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79678.log.gz 2024-09-18T00:42:31.395 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.142229.log.gz 2024-09-18T00:42:31.395 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.142335.log.gz 2024-09-18T00:42:31.398 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121546.log.gz 2024-09-18T00:42:31.399 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.130343.log.gz 2024-09-18T00:42:31.401 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113072.log.gz 2024-09-18T00:42:31.404 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41903.log.gz 2024-09-18T00:42:31.406 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96409.log.gz 2024-09-18T00:42:31.408 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114291.log.gz 2024-09-18T00:42:31.410 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44006.log.gz 2024-09-18T00:42:31.412 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111016.log.gz 2024-09-18T00:42:31.414 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64993.log.gz 2024-09-18T00:42:31.415 INFO:teuthology.orchestra.run.smithi192.stderr: 66.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82136.log.gz 2024-09-18T00:42:31.416 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69290.log.gz 2024-09-18T00:42:31.417 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149196.log.gz 2024-09-18T00:42:31.419 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111224.log.gz 2024-09-18T00:42:31.421 INFO:teuthology.orchestra.run.smithi192.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112822.log.gz 2024-09-18T00:42:31.423 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113911.log.gz 2024-09-18T00:42:31.425 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107348.log.gz 2024-09-18T00:42:31.427 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.95287.log.gz 2024-09-18T00:42:31.430 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123333.log.gz 2024-09-18T00:42:31.431 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.123330.log.gz 2024-09-18T00:42:31.433 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114542.log.gz 2024-09-18T00:42:31.437 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42324.log.gz 2024-09-18T00:42:31.442 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53615.log.gz 2024-09-18T00:42:31.444 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122090.log.gz 2024-09-18T00:42:31.447 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112063.log.gz 2024-09-18T00:42:31.448 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112068.log.gz 2024-09-18T00:42:31.450 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.96051.log.gz 2024-09-18T00:42:31.460 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59049.log.gz 2024-09-18T00:42:31.460 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59008.log.gz 2024-09-18T00:42:31.464 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150143.log.gz 2024-09-18T00:42:31.467 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60667.log.gz 2024-09-18T00:42:31.469 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67761.log.gz 2024-09-18T00:42:31.473 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111435.log.gz 2024-09-18T00:42:31.476 INFO:teuthology.orchestra.run.smithi192.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.84612.log.gz 2024-09-18T00:42:31.480 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76390.log.gz 2024-09-18T00:42:31.481 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77198.log.gz 2024-09-18T00:42:31.483 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89652.log.gz 2024-09-18T00:42:31.485 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.89740.log.gz 2024-09-18T00:42:31.491 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.99830.log.gz 2024-09-18T00:42:31.493 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.86812.log.gz 2024-09-18T00:42:31.495 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.77430.log.gz 2024-09-18T00:42:31.497 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100964.log.gz 2024-09-18T00:42:31.497 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.151793.log.gz 2024-09-18T00:42:31.507 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50165.log.gz 2024-09-18T00:42:31.510 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138224.log.gz 2024-09-18T00:42:31.515 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73929.log.gz 2024-09-18T00:42:31.516 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.74037.log.gz 2024-09-18T00:42:31.518 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137426.log.gz 2024-09-18T00:42:31.521 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66168.log.gz 2024-09-18T00:42:31.524 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.125702.log.gz 2024-09-18T00:42:31.527 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72649.log.gz 2024-09-18T00:42:31.530 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125409.log.gz 2024-09-18T00:42:31.535 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62726.log.gz 2024-09-18T00:42:31.540 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121567.log.gz 2024-09-18T00:42:31.545 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53222.log.gz 2024-09-18T00:42:31.547 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94975.log.gz 2024-09-18T00:42:31.548 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.106848.log.gz 2024-09-18T00:42:31.548 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.118477.log.gz 2024-09-18T00:42:31.550 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.91715.log.gz 2024-09-18T00:42:31.554 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111226.log.gz 2024-09-18T00:42:31.556 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52027.log.gz 2024-09-18T00:42:31.557 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.141449.log.gz 2024-09-18T00:42:31.561 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54779.log.gz 2024-09-18T00:42:31.564 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.57013.log.gz 2024-09-18T00:42:31.568 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.149533.log.gz 2024-09-18T00:42:31.572 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117866.log.gz 2024-09-18T00:42:31.573 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.130148.log.gz 2024-09-18T00:42:31.575 INFO:teuthology.orchestra.run.smithi192.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.140785.log.gz 2024-09-18T00:42:31.577 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60258.log.gz 2024-09-18T00:42:31.580 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.127537.log.gz 2024-09-18T00:42:31.586 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134425.log.gz 2024-09-18T00:42:31.587 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73824.log.gz 2024-09-18T00:42:31.590 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.135557.log.gz 2024-09-18T00:42:31.592 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137028.log.gz 2024-09-18T00:42:31.595 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74459.log.gz 2024-09-18T00:42:31.597 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.admin.69222.log.gz 2024-09-18T00:42:31.600 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40220.log.gz 2024-09-18T00:42:31.603 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40322.log.gz 2024-09-18T00:42:31.606 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126544.log.gz 2024-09-18T00:42:31.609 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.75172.log.gz 2024-09-18T00:42:31.609 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75211.log.gz 2024-09-18T00:42:31.612 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125954.log.gz 2024-09-18T00:42:31.614 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124780.log.gz 2024-09-18T00:42:31.619 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.151423.log.gz 2024-09-18T00:42:31.620 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.92266.log.gz 2024-09-18T00:42:31.624 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133919.log.gz 2024-09-18T00:42:31.627 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137446.log.gz 2024-09-18T00:42:31.629 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.86036.log.gz 2024-09-18T00:42:31.632 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91188.log.gz 2024-09-18T00:42:31.635 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.101676.log.gz 2024-09-18T00:42:31.637 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76717.log.gz 2024-09-18T00:42:31.641 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.130606.log.gz 2024-09-18T00:42:31.641 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62615.log.gz 2024-09-18T00:42:31.644 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39003.log.gz 2024-09-18T00:42:31.650 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114753.log.gz 2024-09-18T00:42:31.653 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149993.log.gz 2024-09-18T00:42:31.659 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118228.log.gz 2024-09-18T00:42:31.661 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.92578.log.gz 2024-09-18T00:42:31.661 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.141241.log.gz 2024-09-18T00:42:31.663 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113764.log.gz 2024-09-18T00:42:31.665 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113242.log.gz 2024-09-18T00:42:31.669 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122803.log.gz 2024-09-18T00:42:31.671 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.127730.log.gz 2024-09-18T00:42:31.671 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121882.log.gz 2024-09-18T00:42:31.673 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52090.log.gz 2024-09-18T00:42:31.676 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.95201.log.gz 2024-09-18T00:42:31.679 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114165.log.gz 2024-09-18T00:42:31.679 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111223.log.gz 2024-09-18T00:42:31.682 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.123898.log.gz 2024-09-18T00:42:31.684 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.69401.log.gz 2024-09-18T00:42:31.685 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124022.log.gz 2024-09-18T00:42:31.694 INFO:teuthology.orchestra.run.smithi096.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-18T00:42:31.701 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.58385.log.gz 2024-09-18T00:42:31.703 INFO:teuthology.orchestra.run.smithi192.stderr: 66.1% -- replaced with /var/log/ceph/cluster1-client.mirror.81794.log.gz 2024-09-18T00:42:31.706 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71378.log.gz 2024-09-18T00:42:31.711 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148796.log.gz 2024-09-18T00:42:31.714 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111812.log.gz 2024-09-18T00:42:31.716 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95107.log.gz 2024-09-18T00:42:31.719 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89949.log.gz 2024-09-18T00:42:31.721 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.115942.log.gz 2024-09-18T00:42:31.724 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114541.log.gz 2024-09-18T00:42:31.727 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.83158.log.gz 2024-09-18T00:42:31.736 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40759.log.gz 2024-09-18T00:42:31.747 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66924.log.gz 2024-09-18T00:42:31.749 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63417.log.gz 2024-09-18T00:42:31.752 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.57270.log.gz 2024-09-18T00:42:31.752 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.57353.log.gz 2024-09-18T00:42:31.754 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.57917.log.gz 2024-09-18T00:42:31.763 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78525.log.gz 2024-09-18T00:42:31.766 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124610.log.gz 2024-09-18T00:42:31.768 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124576.log.gz 2024-09-18T00:42:31.774 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.admin.93029.log.gz 2024-09-18T00:42:31.779 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107394.log.gz 2024-09-18T00:42:31.784 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.90601.log.gz 2024-09-18T00:42:31.788 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88730.log.gz 2024-09-18T00:42:31.791 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.150503.log.gz 2024-09-18T00:42:31.794 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88181.log.gz 2024-09-18T00:42:31.795 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36533.log.gz 2024-09-18T00:42:31.796 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36659.log.gz 2024-09-18T00:42:31.797 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.36929.log.gz 2024-09-18T00:42:31.798 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37118.log.gz 2024-09-18T00:42:31.800 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102476.log.gz 2024-09-18T00:42:31.805 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39150.log.gz 2024-09-18T00:42:31.807 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43636.log.gz 2024-09-18T00:42:31.811 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.83366.log.gz 2024-09-18T00:42:31.815 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39403.log.gz 2024-09-18T00:42:31.817 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40091.log.gz 2024-09-18T00:42:31.817 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95175.log.gz 2024-09-18T00:42:31.820 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.139740.log.gz 2024-09-18T00:42:31.820 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.139874.log.gz 2024-09-18T00:42:31.823 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122704.log.gz 2024-09-18T00:42:31.825 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112441.log.gz 2024-09-18T00:42:31.827 INFO:teuthology.orchestra.run.smithi192.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.94514.log.gz 2024-09-18T00:42:31.829 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.106426.log.gz 2024-09-18T00:42:31.831 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95420.log.gz 2024-09-18T00:42:31.833 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.104642.log.gz 2024-09-18T00:42:31.836 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112274.log.gz 2024-09-18T00:42:31.839 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.116739.log.gz 2024-09-18T00:42:31.841 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52550.log.gz 2024-09-18T00:42:31.844 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43702.log.gz 2024-09-18T00:42:31.848 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.75572.log.gz 2024-09-18T00:42:31.850 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.131795.log.gz 2024-09-18T00:42:31.853 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.45089.log.gz 2024-09-18T00:42:31.857 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.97782.log.gz 2024-09-18T00:42:31.858 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.91814.log.gz 2024-09-18T00:42:31.862 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42957.log.gz 2024-09-18T00:42:31.865 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.149782.log.gz 2024-09-18T00:42:31.869 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.104049.log.gz 2024-09-18T00:42:31.870 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.104067.log.gz 2024-09-18T00:42:31.870 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104228.log.gz 2024-09-18T00:42:31.874 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.143710.log.gz 2024-09-18T00:42:31.878 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47185.log.gz 2024-09-18T00:42:31.880 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126502.log.gz 2024-09-18T00:42:31.885 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82809.log.gz 2024-09-18T00:42:31.889 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87020.log.gz 2024-09-18T00:42:31.891 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.88394.log.gz 2024-09-18T00:42:31.891 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102057.log.gz 2024-09-18T00:42:31.893 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.99996.log.gz 2024-09-18T00:42:31.895 INFO:teuthology.orchestra.run.smithi192.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36775.log.gz 2024-09-18T00:42:31.895 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77431.log.gz 2024-09-18T00:42:31.901 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87271.log.gz 2024-09-18T00:42:31.904 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.75941.log.gz 2024-09-18T00:42:31.904 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.150536.log.gz 2024-09-18T00:42:31.906 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49772.log.gz 2024-09-18T00:42:31.907 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117752.log.gz 2024-09-18T00:42:31.908 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107993.log.gz 2024-09-18T00:42:31.914 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138054.log.gz 2024-09-18T00:42:31.917 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125559.log.gz 2024-09-18T00:42:31.919 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.139149.log.gz 2024-09-18T00:42:31.924 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66228.log.gz 2024-09-18T00:42:31.925 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66690.log.gz 2024-09-18T00:42:31.929 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.145667.log.gz 2024-09-18T00:42:31.929 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.107263.log.gz 2024-09-18T00:42:31.931 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136904.log.gz 2024-09-18T00:42:31.937 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67112.log.gz 2024-09-18T00:42:31.940 INFO:teuthology.orchestra.run.smithi192.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.94266.log.gz 2024-09-18T00:42:31.942 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.49068.log.gz 2024-09-18T00:42:31.946 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120024.log.gz 2024-09-18T00:42:31.948 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65288.log.gz 2024-09-18T00:42:31.949 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38179.log.gz 2024-09-18T00:42:31.954 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50997.log.gz 2024-09-18T00:42:31.955 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67750.log.gz 2024-09-18T00:42:31.958 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.127330.log.gz 2024-09-18T00:42:31.958 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.149473.log.gz 2024-09-18T00:42:31.963 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64215.log.gz 2024-09-18T00:42:31.967 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63078.log.gz 2024-09-18T00:42:31.970 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63144.log.gz 2024-09-18T00:42:31.975 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53643.log.gz 2024-09-18T00:42:31.977 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60168.log.gz 2024-09-18T00:42:31.978 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.92941.log.gz 2024-09-18T00:42:31.982 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111434.log.gz 2024-09-18T00:42:31.984 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54399.log.gz 2024-09-18T00:42:31.993 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123140.log.gz 2024-09-18T00:42:31.995 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123687.log.gz 2024-09-18T00:42:31.997 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120389.log.gz 2024-09-18T00:42:31.999 INFO:teuthology.orchestra.run.smithi192.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.admin.118068.log.gz 2024-09-18T00:42:32.001 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79594.log.gz 2024-09-18T00:42:32.003 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52740.log.gz 2024-09-18T00:42:32.006 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52888.log.gz 2024-09-18T00:42:32.008 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43962.log.gz 2024-09-18T00:42:32.011 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.122471.log.gz 2024-09-18T00:42:32.013 INFO:teuthology.orchestra.run.smithi192.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46843.log.gz 2024-09-18T00:42:32.015 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75422.log.gz 2024-09-18T00:42:32.020 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135850.log.gz 2024-09-18T00:42:32.023 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66925.log.gz 2024-09-18T00:42:32.024 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.56387.log.gz 2024-09-18T00:42:32.029 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.132176.log.gz 2024-09-18T00:42:32.033 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133962.log.gz 2024-09-18T00:42:32.039 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.142537.log.gz 2024-09-18T00:42:32.042 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.88814.log.gz 2024-09-18T00:42:32.044 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99997.log.gz 2024-09-18T00:42:32.048 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109961.log.gz 2024-09-18T00:42:32.049 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.150414.log.gz 2024-09-18T00:42:32.049 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43073.log.gz 2024-09-18T00:42:32.052 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90974.log.gz 2024-09-18T00:42:32.056 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90284.log.gz 2024-09-18T00:42:32.057 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70945.log.gz 2024-09-18T00:42:32.061 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.83581.log.gz 2024-09-18T00:42:32.068 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66733.log.gz 2024-09-18T00:42:32.069 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111391.log.gz 2024-09-18T00:42:32.072 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.117410.log.gz 2024-09-18T00:42:32.074 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98277.log.gz 2024-09-18T00:42:32.076 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114081.log.gz 2024-09-18T00:42:32.079 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54148.log.gz 2024-09-18T00:42:32.085 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52198.log.gz 2024-09-18T00:42:32.091 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.142619.log.gz 2024-09-18T00:42:32.094 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111643.log.gz 2024-09-18T00:42:32.096 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.97584.log.gz 2024-09-18T00:42:32.098 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41863.log.gz 2024-09-18T00:42:32.102 INFO:teuthology.orchestra.run.smithi192.stderr: 53.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35351.log.gz 2024-09-18T00:42:32.102 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.83233.log.gz 2024-09-18T00:42:32.105 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.80620.log.gz 2024-09-18T00:42:32.109 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101780.log.gz 2024-09-18T00:42:32.110 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64212.log.gz 2024-09-18T00:42:32.112 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67942.log.gz 2024-09-18T00:42:32.116 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75718.log.gz 2024-09-18T00:42:32.121 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.131325.log.gz 2024-09-18T00:42:32.123 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131618.log.gz 2024-09-18T00:42:32.123 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131716.log.gz 2024-09-18T00:42:32.130 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51943.log.gz 2024-09-18T00:42:32.134 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55577.log.gz 2024-09-18T00:42:32.144 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114371.log.gz 2024-09-18T00:42:32.151 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.82789.log.gz 2024-09-18T00:42:32.156 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.47548.log.gz 2024-09-18T00:42:32.158 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73825.log.gz 2024-09-18T00:42:32.162 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64968.log.gz 2024-09-18T00:42:32.166 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.144936.log.gz 2024-09-18T00:42:32.167 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.68378.log.gz 2024-09-18T00:42:32.174 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63794.log.gz 2024-09-18T00:42:32.175 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51189.log.gz 2024-09-18T00:42:32.176 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.58499.log.gz 2024-09-18T00:42:32.179 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38728.log.gz 2024-09-18T00:42:32.179 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.152229.log.gz 2024-09-18T00:42:32.181 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112275.log.gz 2024-09-18T00:42:32.181 INFO:teuthology.orchestra.run.smithi192.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.35215.log.gz 2024-09-18T00:42:32.184 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.130391.log.gz 2024-09-18T00:42:32.185 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38180.log.gz 2024-09-18T00:42:32.188 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136879.log.gz 2024-09-18T00:42:32.192 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90389.log.gz 2024-09-18T00:42:32.194 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76273.log.gz 2024-09-18T00:42:32.197 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100546.log.gz 2024-09-18T00:42:32.203 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108297.log.gz 2024-09-18T00:42:32.208 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100252.log.gz 2024-09-18T00:42:32.212 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.45840.log.gz 2024-09-18T00:42:32.212 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.102222.log.gz 2024-09-18T00:42:32.213 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.127644.log.gz 2024-09-18T00:42:32.218 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.91687.log.gz 2024-09-18T00:42:32.219 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124442.log.gz 2024-09-18T00:42:32.223 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.112486.log.gz 2024-09-18T00:42:32.223 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112485.log.gz 2024-09-18T00:42:32.228 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.116674.log.gz 2024-09-18T00:42:32.228 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105567.log.gz 2024-09-18T00:42:32.230 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140214.log.gz 2024-09-18T00:42:32.235 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41902.log.gz 2024-09-18T00:42:32.237 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.115786.log.gz 2024-09-18T00:42:32.239 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.96429.log.gz 2024-09-18T00:42:32.241 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.56827.log.gz 2024-09-18T00:42:32.242 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.110803.log.gz 2024-09-18T00:42:32.244 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122849.log.gz 2024-09-18T00:42:32.251 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.admin.58868.log.gz 2024-09-18T00:42:32.253 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.119370.log.gz 2024-09-18T00:42:32.256 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.132475.log.gz 2024-09-18T00:42:32.259 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.92193.log.gz 2024-09-18T00:42:32.261 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.92788.log.gz 2024-09-18T00:42:32.266 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.149742.log.gz 2024-09-18T00:42:32.268 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.104417.log.gz 2024-09-18T00:42:32.269 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109922.log.gz 2024-09-18T00:42:32.274 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124273.log.gz 2024-09-18T00:42:32.276 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141750.log.gz 2024-09-18T00:42:32.279 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113745.log.gz 2024-09-18T00:42:32.281 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112483.log.gz 2024-09-18T00:42:32.281 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98167.log.gz 2024-09-18T00:42:32.289 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110511.log.gz 2024-09-18T00:42:32.291 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.92407.log.gz 2024-09-18T00:42:32.294 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.106894.log.gz 2024-09-18T00:42:32.295 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101801.log.gz 2024-09-18T00:42:32.296 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47050.log.gz 2024-09-18T00:42:32.300 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131819.log.gz 2024-09-18T00:42:32.300 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.88036.log.gz 2024-09-18T00:42:32.303 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50912.log.gz 2024-09-18T00:42:32.307 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.133311.log.gz 2024-09-18T00:42:32.310 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42023.log.gz 2024-09-18T00:42:32.311 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.117565.log.gz 2024-09-18T00:42:32.314 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95384.log.gz 2024-09-18T00:42:32.317 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.95249.log.gz 2024-09-18T00:42:32.325 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.130805.log.gz 2024-09-18T00:42:32.330 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145693.log.gz 2024-09-18T00:42:32.331 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133543.log.gz 2024-09-18T00:42:32.334 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.133459.log.gz 2024-09-18T00:42:32.337 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148103.log.gz 2024-09-18T00:42:32.339 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.38057.log.gz 2024-09-18T00:42:32.342 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53179.log.gz 2024-09-18T00:42:32.345 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138291.log.gz 2024-09-18T00:42:32.348 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.87022.log.gz 2024-09-18T00:42:32.350 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98779.log.gz 2024-09-18T00:42:32.353 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101700.log.gz 2024-09-18T00:42:32.353 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36965.log.gz 2024-09-18T00:42:32.355 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.77155.log.gz 2024-09-18T00:42:32.362 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131639.log.gz 2024-09-18T00:42:32.366 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.79089.log.gz 2024-09-18T00:42:32.367 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.95488.log.gz 2024-09-18T00:42:32.367 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89320.log.gz 2024-09-18T00:42:32.370 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78942.log.gz 2024-09-18T00:42:32.374 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126123.log.gz 90.0% 2024-09-18T00:42:32.374 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.126165.log.gz 2024-09-18T00:42:32.376 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.125282.log.gz 2024-09-18T00:42:32.380 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86037.log.gz 2024-09-18T00:42:32.383 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110132.log.gz 2024-09-18T00:42:32.388 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60628.log.gz 2024-09-18T00:42:32.389 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60806.log.gz 2024-09-18T00:42:32.391 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123834.log.gz 2024-09-18T00:42:32.394 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54778.log.gz 2024-09-18T00:42:32.395 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.56482.log.gz 2024-09-18T00:42:32.397 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.80820.log.gz 2024-09-18T00:42:32.399 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.81180.log.gz 2024-09-18T00:42:32.411 INFO:teuthology.orchestra.run.smithi192.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.admin.70557.log.gz 2024-09-18T00:42:32.413 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.admin.70785.log.gz 2024-09-18T00:42:32.416 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112402.log.gz 2024-09-18T00:42:32.419 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48635.log.gz 2024-09-18T00:42:32.421 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59671.log.gz 2024-09-18T00:42:32.431 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.73928.log.gz 2024-09-18T00:42:32.433 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74372.log.gz 2024-09-18T00:42:32.436 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.132743.log.gz 2024-09-18T00:42:32.438 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74581.log.gz 2024-09-18T00:42:32.440 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128449.log.gz 2024-09-18T00:42:32.443 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62724.log.gz 2024-09-18T00:42:32.446 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62448.log.gz 2024-09-18T00:42:32.450 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145037.log.gz 2024-09-18T00:42:32.452 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135744.log.gz 2024-09-18T00:42:32.456 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75002.log.gz 2024-09-18T00:42:32.456 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100253.log.gz 2024-09-18T00:42:32.458 INFO:teuthology.orchestra.run.smithi192.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.admin.105769.log.gz 2024-09-18T00:42:32.459 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125753.log.gz 2024-09-18T00:42:32.466 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.108852.log.gz 2024-09-18T00:42:32.468 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109754.log.gz 2024-09-18T00:42:32.471 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77872.log.gz 2024-09-18T00:42:32.473 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110091.log.gz 2024-09-18T00:42:32.475 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110175.log.gz 2024-09-18T00:42:32.480 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138014.log.gz 2024-09-18T00:42:32.483 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44442.log.gz 2024-09-18T00:42:32.487 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.44731.log.gz 2024-09-18T00:42:32.492 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126209.log.gz 2024-09-18T00:42:32.495 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.65137.log.gz 2024-09-18T00:42:32.499 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.141066.log.gz 2024-09-18T00:42:32.504 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.105200.log.gz 2024-09-18T00:42:32.504 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.105201.log.gz 2024-09-18T00:42:32.504 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42493.log.gz 2024-09-18T00:42:32.507 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41602.log.gz 2024-09-18T00:42:32.509 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123560.log.gz 2024-09-18T00:42:32.511 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117658.log.gz 2024-09-18T00:42:32.515 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98232.log.gz 2024-09-18T00:42:32.518 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107653.log.gz 2024-09-18T00:42:32.519 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79803.log.gz 2024-09-18T00:42:32.526 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59575.log.gz 2024-09-18T00:42:32.526 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58210.log.gz 2024-09-18T00:42:32.527 INFO:teuthology.orchestra.run.smithi192.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.59183.log.gz 2024-09-18T00:42:32.529 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.69643.log.gz 2024-09-18T00:42:32.530 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77768.log.gz 2024-09-18T00:42:32.531 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.93997.log.gz 2024-09-18T00:42:32.533 INFO:teuthology.orchestra.run.smithi192.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143194.log.gz 2024-09-18T00:42:32.539 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.99366.log.gz 2024-09-18T00:42:32.543 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63203.log.gz 2024-09-18T00:42:32.547 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.121335.log.gz 2024-09-18T00:42:32.549 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62617.log.gz 2024-09-18T00:42:32.554 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91423.log.gz 2024-09-18T00:42:32.558 INFO:teuthology.orchestra.run.smithi192.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.36423.log.gz 2024-09-18T00:42:32.561 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64131.log.gz 2024-09-18T00:42:32.562 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64170.log.gz 2024-09-18T00:42:32.562 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64171.log.gz 2024-09-18T00:42:32.568 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39606.log.gz 2024-09-18T00:42:32.570 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39629.log.gz 2024-09-18T00:42:32.571 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39760.log.gz 2024-09-18T00:42:32.576 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40180.log.gz 2024-09-18T00:42:32.579 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137575.log.gz 2024-09-18T00:42:32.582 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78710.log.gz 2024-09-18T00:42:32.587 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72470.log.gz 2024-09-18T00:42:32.591 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102852.log.gz 2024-09-18T00:42:32.593 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100377.log.gz 2024-09-18T00:42:32.595 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77810.log.gz 2024-09-18T00:42:32.595 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.104216.log.gz 2024-09-18T00:42:32.598 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88391.log.gz 2024-09-18T00:42:32.599 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52195.log.gz 2024-09-18T00:42:32.602 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103044.log.gz 2024-09-18T00:42:32.605 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98988.log.gz 2024-09-18T00:42:32.605 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140183.log.gz 2024-09-18T00:42:32.608 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85740.log.gz 2024-09-18T00:42:32.612 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.89800.log.gz 2024-09-18T00:42:32.614 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.152036.log.gz 2024-09-18T00:42:32.618 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66063.log.gz 2024-09-18T00:42:32.621 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134929.log.gz 2024-09-18T00:42:32.628 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55700.log.gz 2024-09-18T00:42:32.631 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.132416.log.gz 2024-09-18T00:42:32.633 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122384.log.gz 2024-09-18T00:42:32.634 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.45349.log.gz 2024-09-18T00:42:32.637 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44151.log.gz 2024-09-18T00:42:32.639 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124200.log.gz 2024-09-18T00:42:32.640 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.118607.log.gz 2024-09-18T00:42:32.642 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.130035.log.gz 2024-09-18T00:42:32.645 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.83472.log.gz 2024-09-18T00:42:32.648 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54022.log.gz 2024-09-18T00:42:32.649 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.94855.log.gz 2024-09-18T00:42:32.651 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.139041.log.gz 2024-09-18T00:42:32.657 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39635.log.gz 2024-09-18T00:42:32.661 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.149995.log.gz 2024-09-18T00:42:32.664 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111181.log.gz 2024-09-18T00:42:32.668 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61992.log.gz 2024-09-18T00:42:32.672 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42116.log.gz 2024-09-18T00:42:32.674 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.84083.log.gz 2024-09-18T00:42:32.676 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144933.log.gz 2024-09-18T00:42:32.676 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68869.log.gz 2024-09-18T00:42:32.681 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52467.log.gz 2024-09-18T00:42:32.685 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71972.log.gz 2024-09-18T00:42:32.687 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72430.log.gz 2024-09-18T00:42:32.688 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73146.log.gz 2024-09-18T00:42:32.689 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60190.log.gz 2024-09-18T00:42:32.694 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53600.log.gz 2024-09-18T00:42:32.698 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63336.log.gz 2024-09-18T00:42:32.698 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63375.log.gz 2024-09-18T00:42:32.703 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57479.log.gz 2024-09-18T00:42:32.705 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55997.log.gz 2024-09-18T00:42:32.710 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36684.log.gz 2024-09-18T00:42:32.712 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38727.log.gz 2024-09-18T00:42:32.715 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51626.log.gz 2024-09-18T00:42:32.718 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98862.log.gz 2024-09-18T00:42:32.720 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99661.log.gz 2024-09-18T00:42:32.723 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.84328.log.gz 2024-09-18T00:42:32.723 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110131.log.gz 2024-09-18T00:42:32.725 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.104587.log.gz 2024-09-18T00:42:32.727 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90511.log.gz 2024-09-18T00:42:32.729 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101847.log.gz 2024-09-18T00:42:32.732 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51188.log.gz 2024-09-18T00:42:32.735 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.135766.log.gz 2024-09-18T00:42:32.737 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78103.log.gz 2024-09-18T00:42:32.741 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125216.log.gz 2024-09-18T00:42:32.744 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89001.log.gz 2024-09-18T00:42:32.747 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88731.log.gz 2024-09-18T00:42:32.751 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.108142.log.gz 2024-09-18T00:42:32.753 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77744.log.gz 2024-09-18T00:42:32.756 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126081.log.gz 2024-09-18T00:42:32.757 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.45696.log.gz 2024-09-18T00:42:32.757 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.95038.log.gz 2024-09-18T00:42:32.763 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.admin.129606.log.gz 2024-09-18T00:42:32.767 INFO:teuthology.orchestra.run.smithi192.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.115306.log.gz 2024-09-18T00:42:32.769 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.70644.log.gz 2024-09-18T00:42:32.770 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124108.log.gz 2024-09-18T00:42:32.773 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55405.log.gz 2024-09-18T00:42:32.777 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114711.log.gz 2024-09-18T00:42:32.777 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51942.log.gz 2024-09-18T00:42:32.780 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71080.log.gz 2024-09-18T00:42:32.785 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.119937.log.gz 2024-09-18T00:42:32.787 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120233.log.gz 2024-09-18T00:42:32.790 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120491.log.gz 2024-09-18T00:42:32.795 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111644.log.gz 2024-09-18T00:42:32.798 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.96013.log.gz 2024-09-18T00:42:32.806 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.143670.log.gz 2024-09-18T00:42:32.810 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74416.log.gz 2024-09-18T00:42:32.813 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67530.log.gz 2024-09-18T00:42:32.814 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.70188.log.gz 2024-09-18T00:42:32.820 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.57653.log.gz 2024-09-18T00:42:32.824 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63416.log.gz 2024-09-18T00:42:32.827 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38940.log.gz 2024-09-18T00:42:32.830 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38726.log.gz 2024-09-18T00:42:32.831 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35835.log.gz 2024-09-18T00:42:32.834 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64910.log.gz 2024-09-18T00:42:32.838 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.47320.log.gz 2024-09-18T00:42:32.841 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133394.log.gz 2024-09-18T00:42:32.843 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135430.log.gz 2024-09-18T00:42:32.846 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124719.log.gz 2024-09-18T00:42:32.847 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112276.log.gz 2024-09-18T00:42:32.851 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.90013.log.gz 2024-09-18T00:42:32.852 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.102142.log.gz 2024-09-18T00:42:32.855 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.110385.log.gz 2024-09-18T00:42:32.858 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76569.log.gz 2024-09-18T00:42:32.864 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.101256.log.gz 2024-09-18T00:42:32.864 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78145.log.gz 2024-09-18T00:42:32.870 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87320.log.gz 2024-09-18T00:42:32.873 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41233.log.gz 2024-09-18T00:42:32.876 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108163.log.gz 2024-09-18T00:42:32.879 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76296.log.gz 2024-09-18T00:42:32.882 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98652.log.gz 2024-09-18T00:42:32.885 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.100819.log.gz 2024-09-18T00:42:32.887 INFO:teuthology.orchestra.run.smithi192.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115766.log.gz 2024-09-18T00:42:32.891 INFO:teuthology.orchestra.run.smithi192.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45881.log.gz 2024-09-18T00:42:32.894 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71946.log.gz 2024-09-18T00:42:32.896 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37253.log.gz 2024-09-18T00:42:32.901 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.116362.log.gz 2024-09-18T00:42:32.903 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.141941.log.gz 2024-09-18T00:42:32.903 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.142119.log.gz 2024-09-18T00:42:32.906 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61173.log.gz 2024-09-18T00:42:32.908 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54358.log.gz 2024-09-18T00:42:32.910 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.56557.log.gz 2024-09-18T00:42:32.913 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97647.log.gz 2024-09-18T00:42:32.915 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60766.log.gz 2024-09-18T00:42:32.918 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122470.log.gz 2024-09-18T00:42:32.918 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.94628.log.gz 2024-09-18T00:42:32.919 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39759.log.gz 2024-09-18T00:42:32.920 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39970.log.gz 2024-09-18T00:42:32.923 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122299.log.gz 2024-09-18T00:42:32.925 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124358.log.gz 2024-09-18T00:42:32.927 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121379.log.gz 2024-09-18T00:42:32.928 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75003.log.gz 2024-09-18T00:42:32.929 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97924.log.gz 2024-09-18T00:42:32.933 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120841.log.gz 2024-09-18T00:42:32.935 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41486.log.gz 2024-09-18T00:42:32.937 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.105108.log.gz 2024-09-18T00:42:32.942 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51669.log.gz 2024-09-18T00:42:32.944 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98630.log.gz 2024-09-18T00:42:32.949 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.77240.log.gz 2024-09-18T00:42:32.953 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110387.log.gz 2024-09-18T00:42:32.954 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46729.log.gz 2024-09-18T00:42:32.957 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136020.log.gz 2024-09-18T00:42:32.958 INFO:teuthology.orchestra.run.smithi192.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130959.log.gz 2024-09-18T00:42:32.959 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.131216.log.gz 2024-09-18T00:42:32.959 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.141961.log.gz 2024-09-18T00:42:32.963 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.132837.log.gz 2024-09-18T00:42:32.966 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110597.log.gz 2024-09-18T00:42:32.966 INFO:teuthology.orchestra.run.smithi192.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.35522.log.gz 2024-09-18T00:42:32.968 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.80177.log.gz 2024-09-18T00:42:32.970 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88182.log.gz 2024-09-18T00:42:32.971 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.92194.log.gz 2024-09-18T00:42:32.972 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.79994.log.gz 2024-09-18T00:42:32.975 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76194.log.gz 2024-09-18T00:42:32.976 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.105205.log.gz 2024-09-18T00:42:32.977 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.139632.log.gz 2024-09-18T00:42:32.978 INFO:teuthology.orchestra.run.smithi192.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.115601.log.gz 2024-09-18T00:42:32.980 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.147371.log.gz 2024-09-18T00:42:32.984 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124823.log.gz 2024-09-18T00:42:32.986 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136817.log.gz 2024-09-18T00:42:32.990 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63917.log.gz 2024-09-18T00:42:32.991 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.65472.log.gz 2024-09-18T00:42:32.993 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50163.log.gz 2024-09-18T00:42:32.996 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64674.log.gz 2024-09-18T00:42:32.998 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64739.log.gz 2024-09-18T00:42:33.001 INFO:teuthology.orchestra.run.smithi192.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-client.admin.82411.log.gz 2024-09-18T00:42:33.002 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83091.log.gz 2024-09-18T00:42:33.005 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64255.log.gz 2024-09-18T00:42:33.015 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.96097.log.gz 2024-09-18T00:42:33.018 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42325.log.gz 2024-09-18T00:42:33.019 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.115453.log.gz 2024-09-18T00:42:33.020 INFO:teuthology.orchestra.run.smithi192.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.143041.log.gz 2024-09-18T00:42:33.021 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.129693.log.gz 2024-09-18T00:42:33.023 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48482.log.gz 2024-09-18T00:42:33.026 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.103588.log.gz 2024-09-18T00:42:33.028 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.69404.log.gz 2024-09-18T00:42:33.032 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54062.log.gz 2024-09-18T00:42:33.035 INFO:teuthology.orchestra.run.smithi192.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.35232.log.gz 2024-09-18T00:42:33.038 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112067.log.gz 2024-09-18T00:42:33.041 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61488.log.gz 2024-09-18T00:42:33.043 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.106084.log.gz 2024-09-18T00:42:33.043 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.106198.log.gz 2024-09-18T00:42:33.046 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.117001.log.gz 2024-09-18T00:42:33.052 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112697.log.gz 2024-09-18T00:42:33.074 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91617.log.gz 2024-09-18T00:42:33.076 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76609.log.gz 2024-09-18T00:42:33.081 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.108094.log.gz 2024-09-18T00:42:33.081 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89950.log.gz 2024-09-18T00:42:33.085 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.99537.log.gz 2024-09-18T00:42:33.088 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100123.log.gz 2024-09-18T00:42:33.090 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.107921.log.gz 2024-09-18T00:42:33.094 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.87703.log.gz 2024-09-18T00:42:33.097 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.47770.log.gz 2024-09-18T00:42:33.101 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.135641.log.gz 2024-09-18T00:42:33.103 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.67448.log.gz 2024-09-18T00:42:33.105 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.142035.log.gz 2024-09-18T00:42:33.106 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89650.log.gz 2024-09-18T00:42:33.111 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78880.log.gz 2024-09-18T00:42:33.115 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39151.log.gz 2024-09-18T00:42:33.120 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.139055.log.gz 2024-09-18T00:42:33.124 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50702.log.gz 2024-09-18T00:42:33.128 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43703.log.gz 2024-09-18T00:42:33.129 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51944.log.gz 2024-09-18T00:42:33.131 INFO:teuthology.orchestra.run.smithi192.stderr: 66.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130129.log.gz 2024-09-18T00:42:33.132 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.130320.log.gz 2024-09-18T00:42:33.132 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.130475.log.gz 2024-09-18T00:42:33.134 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.92483.log.gz 2024-09-18T00:42:33.136 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123224.log.gz 2024-09-18T00:42:33.140 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110722.log.gz 2024-09-18T00:42:33.143 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114332.log.gz 2024-09-18T00:42:33.144 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.93971.log.gz 2024-09-18T00:42:33.146 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111648.log.gz 2024-09-18T00:42:33.148 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.83888.log.gz 2024-09-18T00:42:33.151 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71828.log.gz 2024-09-18T00:42:33.153 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.117585.log.gz 2024-09-18T00:42:33.155 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60341.log.gz 2024-09-18T00:42:33.156 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61839.log.gz 2024-09-18T00:42:33.157 INFO:teuthology.orchestra.run.smithi192.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.107160.log.gz 2024-09-18T00:42:33.160 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.56445.log.gz 2024-09-18T00:42:33.165 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86584.log.gz 2024-09-18T00:42:33.167 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87141.log.gz 2024-09-18T00:42:33.169 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141428.log.gz 2024-09-18T00:42:33.170 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54819.log.gz 2024-09-18T00:42:33.174 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111436.log.gz 2024-09-18T00:42:33.175 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70831.log.gz 2024-09-18T00:42:33.177 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.103612.log.gz 2024-09-18T00:42:33.177 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.103723.log.gz 2024-09-18T00:42:33.182 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104432.log.gz 2024-09-18T00:42:33.188 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.113325.log.gz 2024-09-18T00:42:33.189 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113452.log.gz 2024-09-18T00:42:33.191 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114581.log.gz 2024-09-18T00:42:33.195 INFO:teuthology.orchestra.run.smithi192.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130690.log.gz 2024-09-18T00:42:33.199 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.142645.log.gz 2024-09-18T00:42:33.202 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.151001.log.gz 2024-09-18T00:42:33.209 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.86753.log.gz 2024-09-18T00:42:33.211 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102562.log.gz 2024-09-18T00:42:33.214 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109968.log.gz 2024-09-18T00:42:33.216 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.146742.log.gz 2024-09-18T00:42:33.220 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40390.log.gz 2024-09-18T00:42:33.222 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84129.log.gz 2024-09-18T00:42:33.227 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77684.log.gz 2024-09-18T00:42:33.231 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101911.log.gz 2024-09-18T00:42:33.231 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104507.log.gz 2024-09-18T00:42:33.235 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58552.log.gz 2024-09-18T00:42:33.235 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.118496.log.gz 2024-09-18T00:42:33.236 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.115604.log.gz 2024-09-18T00:42:33.242 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112611.log.gz 2024-09-18T00:42:33.243 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53559.log.gz 2024-09-18T00:42:33.250 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51710.log.gz 2024-09-18T00:42:33.255 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.65475.log.gz 2024-09-18T00:42:33.256 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39422.log.gz 2024-09-18T00:42:33.258 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.95898.log.gz 2024-09-18T00:42:33.263 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52847.log.gz 2024-09-18T00:42:33.266 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111864.log.gz 2024-09-18T00:42:33.266 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41272.log.gz 2024-09-18T00:42:33.267 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.46863.log.gz 2024-09-18T00:42:33.272 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63667.log.gz 2024-09-18T00:42:33.272 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.128209.log.gz 2024-09-18T00:42:33.277 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64590.log.gz 2024-09-18T00:42:33.282 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72704.log.gz 2024-09-18T00:42:33.286 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49471.log.gz 2024-09-18T00:42:33.289 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.130652.log.gz 2024-09-18T00:42:33.291 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.135619.log.gz 2024-09-18T00:42:33.295 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.admin.47233.log.gz 2024-09-18T00:42:33.298 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136964.log.gz 2024-09-18T00:42:33.303 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.146449.log.gz 2024-09-18T00:42:33.307 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.101510.log.gz 2024-09-18T00:42:33.313 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85303.log.gz 2024-09-18T00:42:33.316 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88314.log.gz 2024-09-18T00:42:33.321 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.87415.log.gz 2024-09-18T00:42:33.324 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88393.log.gz 2024-09-18T00:42:33.324 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110171.log.gz 2024-09-18T00:42:33.325 INFO:teuthology.orchestra.run.smithi192.stderr: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35573.log.gz 2024-09-18T00:42:33.328 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109881.log.gz 2024-09-18T00:42:33.332 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135221.log.gz 2024-09-18T00:42:33.335 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.150804.log.gz 2024-09-18T00:42:33.337 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50955.log.gz 2024-09-18T00:42:33.341 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54992.log.gz 2024-09-18T00:42:33.347 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138373.log.gz 2024-09-18T00:42:33.347 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.44942.log.gz 2024-09-18T00:42:33.350 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78733.log.gz 2024-09-18T00:42:33.353 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51836.log.gz 2024-09-18T00:42:33.353 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.79898.log.gz 2024-09-18T00:42:33.354 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51879.log.gz 2024-09-18T00:42:33.357 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.82270.log.gz 2024-09-18T00:42:33.359 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82849.log.gz 2024-09-18T00:42:33.366 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60496.log.gz 2024-09-18T00:42:33.369 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67846.log.gz 2024-09-18T00:42:38.306 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.36532.log.gz 2024-09-18T00:42:40.749 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.36837.log.gz 2024-09-18T00:42:40.764 INFO:teuthology.orchestra.run.smithi192.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83234.log.gz 2024-09-18T00:42:40.764 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113327.log.gz 2024-09-18T00:42:40.764 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.89880.log.gz 2024-09-18T00:42:40.765 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52655.log.gz 2024-09-18T00:42:40.765 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62324.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.54736.log.gz 2024-09-18T00:42:40.765 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:40.765 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64738.log.gz 2024-09-18T00:42:40.765 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.77156.log.gz 2024-09-18T00:42:40.766 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123939.log.gz 2024-09-18T00:42:40.766 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.6% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88709.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.91062.log.gz 2024-09-18T00:42:40.766 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:40.767 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.8% 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.67194.log.gz 2024-09-18T00:42:40.767 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125661.log.gz 2024-09-18T00:42:40.768 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.6% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37999.log.gz 2024-09-18T00:42:40.768 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.72856.log.gz 2024-09-18T00:42:40.768 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.74268.log.gz 2024-09-18T00:42:40.768 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.147687.log.gz 2024-09-18T00:42:40.769 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.149636.log.gz 2024-09-18T00:42:40.769 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137468.log.gz 2024-09-18T00:42:40.771 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.134149.log.gz 2024-09-18T00:42:40.771 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143589.log.gz 2024-09-18T00:42:40.772 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67197.log.gz 2024-09-18T00:42:40.772 INFO:teuthology.orchestra.run.smithi192.stderr: 54.4% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.143475.log.gz 2024-09-18T00:42:40.772 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.83823.log.gz 2024-09-18T00:42:40.772 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43163.log.gz 2024-09-18T00:42:40.772 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63709.log.gz 2024-09-18T00:42:40.772 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.148736.log.gz 2024-09-18T00:42:40.772 INFO:teuthology.orchestra.run.smithi192.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54526.log.gz 2024-09-18T00:42:40.772 INFO:teuthology.orchestra.run.smithi192.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78794.log.gz 2024-09-18T00:42:40.772 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66166.log.gz 2024-09-18T00:42:40.772 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101846.log.gz 2024-09-18T00:42:40.772 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62725.log.gz 2024-09-18T00:42:40.772 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.95336.log.gz 2024-09-18T00:42:40.772 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.142515.log.gz 2024-09-18T00:42:40.773 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102309.log.gz 2024-09-18T00:42:40.773 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.118955.log.gz 2024-09-18T00:42:40.773 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.134110.log.gz 2024-09-18T00:42:40.773 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122553.log.gz 2024-09-18T00:42:40.773 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.124109.log.gz 2024-09-18T00:42:40.775 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.88120.log.gz 2024-09-18T00:42:40.775 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.74836.log.gz 2024-09-18T00:42:40.775 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.106312.log.gz 2024-09-18T00:42:40.775 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74229.log.gz 2024-09-18T00:42:40.775 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102267.log.gz 2024-09-18T00:42:40.775 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.149261.log.gz 2024-09-18T00:42:40.775 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121880.log.gz 2024-09-18T00:42:40.775 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.152254.log.gz 2024-09-18T00:42:40.775 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124065.log.gz 2024-09-18T00:42:40.775 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.107853.log.gz 2024-09-18T00:42:40.775 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.146322.log.gz 2024-09-18T00:42:40.776 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.146423.log.gz 2024-09-18T00:42:40.776 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60260.log.gz 2024-09-18T00:42:40.776 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.86518.log.gz 2024-09-18T00:42:40.777 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138807.log.gz 2024-09-18T00:42:40.777 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.144910.log.gz 2024-09-18T00:42:40.778 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% 90.7% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.145016.log.gz 2024-09-18T00:42:40.778 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43166.log.gz 2024-09-18T00:42:40.778 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.131737.log.gz 2024-09-18T00:42:40.778 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90743.log.gz 2024-09-18T00:42:40.778 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63942.log.gz 2024-09-18T00:42:40.778 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.83368.log.gz 2024-09-18T00:42:40.778 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98361.log.gz 2024-09-18T00:42:40.778 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88813.log.gz 2024-09-18T00:42:40.778 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50343.log.gz 2024-09-18T00:42:40.778 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44102.log.gz 2024-09-18T00:42:40.782 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.99470.log.gz 2024-09-18T00:42:40.782 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69309.log.gz 2024-09-18T00:42:40.782 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49292.log.gz 2024-09-18T00:42:40.783 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.77659.log.gz 2024-09-18T00:42:40.783 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53832.log.gz 2024-09-18T00:42:40.783 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.143192.log.gz 2024-09-18T00:42:40.783 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98152.log.gz 2024-09-18T00:42:40.783 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.35488.log.gz 2024-09-18T00:42:40.783 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.151153.log.gz 2024-09-18T00:42:40.783 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50473.log.gz 2024-09-18T00:42:40.783 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59803.log.gz 2024-09-18T00:42:40.783 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.admin.128444.log.gz 2024-09-18T00:42:40.783 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124946.log.gz 2024-09-18T00:42:40.783 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135494.log.gz 2024-09-18T00:42:40.783 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86752.log.gz 2024-09-18T00:42:40.783 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.59096.log.gz 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87163.log.gz 2024-09-18T00:42:40.783 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:40.784 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.87620.log.gz 2024-09-18T00:42:40.784 INFO:teuthology.orchestra.run.smithi192.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58251.log.gz 2024-09-18T00:42:40.784 INFO:teuthology.orchestra.run.smithi192.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.81908.log.gz 2024-09-18T00:42:40.784 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.admin.58640.log.gz 2024-09-18T00:42:40.784 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138080.log.gz 2024-09-18T00:42:40.784 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64004.log.gz 2024-09-18T00:42:40.784 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.149994.log.gz 2024-09-18T00:42:40.784 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.98883.log.gz 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.68477.log.gz 2024-09-18T00:42:40.784 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:40.784 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.95535.log.gz 2024-09-18T00:42:40.784 INFO:teuthology.orchestra.run.smithi192.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.80260.log.gz 2024-09-18T00:42:40.784 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.92657.log.gz 2024-09-18T00:42:40.785 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76465.log.gz 2024-09-18T00:42:40.785 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39863.log.gz 2024-09-18T00:42:40.785 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42113.log.gz 2024-09-18T00:42:40.785 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53056.log.gz 2024-09-18T00:42:40.785 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111437.log.gz 2024-09-18T00:42:40.798 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.0% 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46707.log.gz 2024-09-18T00:42:40.799 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66421.log.gz 2024-09-18T00:42:40.799 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79471.log.gz 2024-09-18T00:42:40.799 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.80627.log.gz 2024-09-18T00:42:40.799 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.66398.log.gz 2024-09-18T00:42:40.799 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% 90.4% 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133054.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.40431.log.gz 2024-09-18T00:42:40.799 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:40.799 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.52973.log.gz 2024-09-18T00:42:40.799 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47507.log.gz 2024-09-18T00:42:40.799 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46936.log.gz 2024-09-18T00:42:40.799 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135431.log.gz 2024-09-18T00:42:40.799 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.94815.log.gz 2024-09-18T00:42:40.800 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.143709.log.gz 2024-09-18T00:42:40.800 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58230.log.gz 2024-09-18T00:42:40.800 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.107196.log.gz 2024-09-18T00:42:40.800 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.48131.log.gz 2024-09-18T00:42:40.800 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130559.log.gz 2024-09-18T00:42:40.800 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.45742.log.gz 2024-09-18T00:42:40.800 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61418.log.gz 2024-09-18T00:42:40.800 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.74203.log.gz 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78374.log.gz 2024-09-18T00:42:40.800 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:40.800 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76424.log.gz 2024-09-18T00:42:40.800 INFO:teuthology.orchestra.run.smithi192.stderr: 2.3% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.77116.log.gz 2024-09-18T00:42:40.800 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124382.log.gz 2024-09-18T00:42:40.800 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.48216.log.gz 2024-09-18T00:42:40.801 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50407.log.gz 2024-09-18T00:42:40.801 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.147116.log.gz 2024-09-18T00:42:40.801 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.146114.log.gz 2024-09-18T00:42:40.801 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.118775.log.gz 2024-09-18T00:42:40.801 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.129036.log.gz 2024-09-18T00:42:40.801 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70147.log.gz 2024-09-18T00:42:40.801 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.148943.log.gz 2024-09-18T00:42:40.801 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.57385.log.gz 2024-09-18T00:42:40.801 INFO:teuthology.orchestra.run.smithi192.stderr: 89.7% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39402.log.gz 2024-09-18T00:42:40.801 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.147745.log.gz 2024-09-18T00:42:40.801 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120390.log.gz 2024-09-18T00:42:40.801 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60018.log.gz 2024-09-18T00:42:40.801 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.149873.log.gz 2024-09-18T00:42:40.802 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.0% 90.7% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79322.log.gz 2024-09-18T00:42:40.802 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101908.log.gz 2024-09-18T00:42:40.802 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62575.log.gz 2024-09-18T00:42:40.802 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67813.log.gz 2024-09-18T00:42:40.802 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137660.log.gz 2024-09-18T00:42:40.802 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.77366.log.gz 2024-09-18T00:42:40.802 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110341.log.gz 2024-09-18T00:42:40.802 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123624.log.gz 2024-09-18T00:42:40.802 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.149474.log.gz 2024-09-18T00:42:40.802 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.57249.log.gz 2024-09-18T00:42:40.802 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78328.log.gz 2024-09-18T00:42:40.802 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.124781.log.gz 2024-09-18T00:42:40.802 INFO:teuthology.orchestra.run.smithi192.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.68236.log.gz 2024-09-18T00:42:40.802 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.97814.log.gz 2024-09-18T00:42:40.803 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55512.log.gz 2024-09-18T00:42:40.803 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.137110.log.gz 2024-09-18T00:42:40.803 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.145645.log.gz 2024-09-18T00:42:40.803 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149263.log.gz 2024-09-18T00:42:40.803 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121670.log.gz 2024-09-18T00:42:40.803 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.134173.log.gz 2024-09-18T00:42:40.803 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72367.log.gz 2024-09-18T00:42:40.803 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77032.log.gz 2024-09-18T00:42:40.803 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.80035.log.gz 2024-09-18T00:42:40.803 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50746.log.gz 2024-09-18T00:42:40.803 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.137112.log.gz 2024-09-18T00:42:40.803 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66314.log.gz 2024-09-18T00:42:40.803 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66774.log.gz 2024-09-18T00:42:40.804 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.124066.log.gz 90.1% 2024-09-18T00:42:40.804 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.54272.log.gz 2024-09-18T00:42:40.804 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58365.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.126954.log.gz 2024-09-18T00:42:40.804 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:40.804 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136972.log.gz 2024-09-18T00:42:40.804 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.132745.log.gz 2024-09-18T00:42:40.804 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52784.log.gz 2024-09-18T00:42:40.804 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77262.log.gz 2024-09-18T00:42:40.804 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37384.log.gz 2024-09-18T00:42:40.804 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.83404.log.gz 2024-09-18T00:42:40.804 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.127916.log.gz 2024-09-18T00:42:40.804 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.89817.log.gz 2024-09-18T00:42:40.805 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53013.log.gz 2024-09-18T00:42:40.805 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41696.log.gz 2024-09-18T00:42:40.805 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51586.log.gz 2024-09-18T00:42:41.073 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.146446.log.gz 2024-09-18T00:42:41.073 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113954.log.gz 2024-09-18T00:42:41.074 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.61174.log.gz 2024-09-18T00:42:41.074 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.149365.log.gz 2024-09-18T00:42:41.074 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.80104.log.gz 2024-09-18T00:42:41.074 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.124039.log.gz 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.123307.log.gz 2024-09-18T00:42:41.074 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.074 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59953.log.gz 2024-09-18T00:42:41.074 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.140926.log.gz 2024-09-18T00:42:41.074 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77811.log.gz 2024-09-18T00:42:41.074 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41064.log.gz 2024-09-18T00:42:41.074 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.124863.log.gz 2024-09-18T00:42:41.074 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.149322.log.gz 2024-09-18T00:42:41.074 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.78944.log.gz 2024-09-18T00:42:41.075 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42322.log.gz 2024-09-18T00:42:41.075 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132476.log.gz 2024-09-18T00:42:41.075 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125850.log.gz 2024-09-18T00:42:41.075 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94701.log.gz 2024-09-18T00:42:41.075 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.99073.log.gz 2024-09-18T00:42:41.075 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.85637.log.gz 2024-09-18T00:42:41.075 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.106653.log.gz 2024-09-18T00:42:41.075 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37082.log.gz 91.4% 2024-09-18T00:42:41.075 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.116615.log.gz 2024-09-18T00:42:41.075 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109643.log.gz 2024-09-18T00:42:41.075 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.89719.log.gz 2024-09-18T00:42:41.075 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59955.log.gz 2024-09-18T00:42:41.075 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39864.log.gz 2024-09-18T00:42:41.076 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.104988.log.gz 2024-09-18T00:42:41.076 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.8% 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.83431.log.gz 2024-09-18T00:42:41.076 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59296.log.gz 2024-09-18T00:42:41.076 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.78416.log.gz 2024-09-18T00:42:41.076 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.106764.log.gz 2024-09-18T00:42:41.076 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.83734.log.gz 2024-09-18T00:42:41.076 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39550.log.gz 2024-09-18T00:42:41.076 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% 89.9% 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.95772.log.gz 2024-09-18T00:42:41.076 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.45451.log.gz 2024-09-18T00:42:41.076 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.92537.log.gz 2024-09-18T00:42:41.076 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67365.log.gz 2024-09-18T00:42:41.076 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107222.log.gz 2024-09-18T00:42:41.076 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120952.log.gz 2024-09-18T00:42:41.077 INFO:teuthology.orchestra.run.smithi192.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.118739.log.gz 2024-09-18T00:42:41.077 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40932.log.gz 2024-09-18T00:42:41.077 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63498.log.gz 2024-09-18T00:42:41.077 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62408.log.gz 2024-09-18T00:42:41.077 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.44889.log.gz 2024-09-18T00:42:41.077 INFO:teuthology.orchestra.run.smithi192.stderr: 66.6% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76842.log.gz 2024-09-18T00:42:41.077 INFO:teuthology.orchestra.run.smithi192.stderr: 66.3% 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.92096.log.gz 2024-09-18T00:42:41.077 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.107985.log.gz 2024-09-18T00:42:41.077 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.72811.log.gz 2024-09-18T00:42:41.077 INFO:teuthology.orchestra.run.smithi192.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55516.log.gz 2024-09-18T00:42:41.077 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.47661.log.gz 2024-09-18T00:42:41.077 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% 80.1% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.89184.log.gz 2024-09-18T00:42:41.078 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44103.log.gz 2024-09-18T00:42:41.078 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.39969.log.gz 2024-09-18T00:42:41.078 INFO:teuthology.orchestra.run.smithi192.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.mirror.141924.log.gz 2024-09-18T00:42:41.078 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87162.log.gz 2024-09-18T00:42:41.078 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109309.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.148844.log.gz 2024-09-18T00:42:41.078 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54231.log.gz 2024-09-18T00:42:41.078 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.078 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101216.log.gz 2024-09-18T00:42:41.078 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.45844.log.gz 2024-09-18T00:42:41.078 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.148506.log.gz 2024-09-18T00:42:41.078 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65832.log.gz 2024-09-18T00:42:41.078 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.134424.log.gz 2024-09-18T00:42:41.078 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39129.log.gz 2024-09-18T00:42:41.079 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.84214.log.gz 2024-09-18T00:42:41.079 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.99409.log.gz 2024-09-18T00:42:41.079 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67320.log.gz 2024-09-18T00:42:41.079 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148945.log.gz 2024-09-18T00:42:41.079 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.admin.92073.log.gz 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86224.log.gz 2024-09-18T00:42:41.079 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.079 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.93857.log.gz 2024-09-18T00:42:41.079 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67069.log.gz 2024-09-18T00:42:41.079 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.72575.log.gz 2024-09-18T00:42:41.079 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.116430.log.gz 2024-09-18T00:42:41.079 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.48851.log.gz 2024-09-18T00:42:41.079 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149323.log.gz 2024-09-18T00:42:41.079 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.104179.log.gz 2024-09-18T00:42:41.080 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 91.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96365.log.gz 2024-09-18T00:42:41.080 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.68226.log.gz 2024-09-18T00:42:41.080 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.47300.log.gz 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.151485.log.gz 2024-09-18T00:42:41.080 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.080 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% 80.3% 79.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72063.log.gz 2024-09-18T00:42:41.080 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.139284.log.gz 2024-09-18T00:42:41.080 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47414.log.gz 2024-09-18T00:42:41.080 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.admin.104853.log.gz 2024-09-18T00:42:41.080 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.142888.log.gz 2024-09-18T00:42:41.080 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.80499.log.gz 2024-09-18T00:42:41.080 INFO:teuthology.orchestra.run.smithi192.stderr: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.35300.log.gz 2024-09-18T00:42:41.080 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101173.log.gz 2024-09-18T00:42:41.080 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125180.log.gz 2024-09-18T00:42:41.081 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.111894.log.gz 2024-09-18T00:42:41.081 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.48284.log.gz 2024-09-18T00:42:41.081 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.113321.log.gz 2024-09-18T00:42:41.081 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.87459.log.gz 2024-09-18T00:42:41.081 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67278.log.gz 2024-09-18T00:42:41.081 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95446.log.gz 2024-09-18T00:42:41.081 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.125408.log.gz 2024-09-18T00:42:41.081 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.119084.log.gz 2024-09-18T00:42:41.081 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60939.log.gz 2024-09-18T00:42:41.081 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48242.log.gz 2024-09-18T00:42:41.081 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107221.log.gz 2024-09-18T00:42:41.081 INFO:teuthology.orchestra.run.smithi192.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142431.log.gz 2024-09-18T00:42:41.081 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39420.log.gz 2024-09-18T00:42:41.082 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.68080.log.gz 2024-09-18T00:42:41.082 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.40487.log.gz 2024-09-18T00:42:41.082 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.95686.log.gz 2024-09-18T00:42:41.082 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.103741.log.gz 2024-09-18T00:42:41.082 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100074.log.gz 2024-09-18T00:42:41.082 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.103763.log.gz 2024-09-18T00:42:41.082 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.44858.log.gz 2024-09-18T00:42:41.082 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.46634.log.gz 2024-09-18T00:42:41.082 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.116094.log.gz 2024-09-18T00:42:41.082 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.125324.log.gz 90.0% 2024-09-18T00:42:41.082 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.102141.log.gz 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.75509.log.gz 2024-09-18T00:42:41.082 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.81058.log.gz 2024-09-18T00:42:41.082 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.139519.log.gz 2024-09-18T00:42:41.083 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.083 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107617.log.gz 2024-09-18T00:42:41.083 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47030.log.gz 2024-09-18T00:42:41.083 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126792.log.gz 2024-09-18T00:42:41.083 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59405.log.gz 2024-09-18T00:42:41.083 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.130807.log.gz 2024-09-18T00:42:41.083 INFO:teuthology.orchestra.run.smithi192.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.71013.log.gz 2024-09-18T00:42:41.083 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.35915.log.gz 2024-09-18T00:42:41.083 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59801.log.gz 2024-09-18T00:42:41.083 INFO:teuthology.orchestra.run.smithi192.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130654.log.gz 2024-09-18T00:42:41.083 INFO:teuthology.orchestra.run.smithi192.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.83093.log.gz 2024-09-18T00:42:41.083 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46614.log.gz 2024-09-18T00:42:41.083 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59407.log.gz 2024-09-18T00:42:41.083 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.35283.log.gz 2024-09-18T00:42:41.084 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.152022.log.gz 2024-09-18T00:42:41.084 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38058.log.gz 2024-09-18T00:42:41.084 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148842.log.gz 2024-09-18T00:42:41.084 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.129009.log.gz 2024-09-18T00:42:41.084 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.99418.log.gz 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.127306.log.gz 2024-09-18T00:42:41.084 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.084 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.148586.log.gz 2024-09-18T00:42:41.084 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.151332.log.gz 2024-09-18T00:42:41.084 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.151168.log.gz 2024-09-18T00:42:41.084 INFO:teuthology.orchestra.run.smithi192.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.83185.log.gz 2024-09-18T00:42:41.084 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129994.log.gz 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.92825.log.gz 2024-09-18T00:42:41.084 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.085 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107547.log.gz 2024-09-18T00:42:41.085 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.57023.log.gz 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.148482.log.gz 2024-09-18T00:42:41.085 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.085 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.143588.log.gz 90.6% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71669.log.gz 2024-09-18T00:42:41.085 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.085 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72277.log.gz 2024-09-18T00:42:41.085 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.127635.log.gz 2024-09-18T00:42:41.085 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41140.log.gz 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42912.log.gz 2024-09-18T00:42:41.085 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.085 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.73886.log.gz 2024-09-18T00:42:41.085 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47968.log.gz 2024-09-18T00:42:41.085 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99199.log.gz 2024-09-18T00:42:41.085 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41063.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.65389.log.gz 2024-09-18T00:42:41.086 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.086 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.117813.log.gz 2024-09-18T00:42:41.086 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105815.log.gz 2024-09-18T00:42:41.086 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.115259.log.gz 2024-09-18T00:42:41.086 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.44882.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.140812.log.gz 2024-09-18T00:42:41.086 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.086 INFO:teuthology.orchestra.run.smithi192.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.82763.log.gz 2024-09-18T00:42:41.086 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42744.log.gz 2024-09-18T00:42:41.086 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.60365.log.gz 2024-09-18T00:42:41.086 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.107312.log.gz 2024-09-18T00:42:41.086 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.139932.log.gz 2024-09-18T00:42:41.086 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 87.7% 67.2% -- replaced with /var/log/ceph/cluster2-client.admin.35471.log.gz 2024-09-18T00:42:41.086 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.33508.log.gz 2024-09-18T00:42:41.086 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.106292.log.gz 2024-09-18T00:42:41.087 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.119891.log.gz 2024-09-18T00:42:41.087 INFO:teuthology.orchestra.run.smithi192.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.82478.log.gz 2024-09-18T00:42:41.087 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118000.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.129652.log.gz 2024-09-18T00:42:41.087 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137174.log.gz 2024-09-18T00:42:41.087 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.087 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.146216.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.115860.log.gz 2024-09-18T00:42:41.087 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.087 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.130523.log.gz 2024-09-18T00:42:41.087 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.93743.log.gz 2024-09-18T00:42:41.087 INFO:teuthology.orchestra.run.smithi192.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.mirror.107769.log.gz 2024-09-18T00:42:41.087 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.46749.log.gz 2024-09-18T00:42:41.087 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71210.log.gz 2024-09-18T00:42:41.088 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111141.log.gz 2024-09-18T00:42:41.088 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.147054.log.gz 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60321.log.gz 2024-09-18T00:42:41.088 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.088 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.130759.log.gz 2024-09-18T00:42:41.088 INFO:teuthology.orchestra.run.smithi192.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142583.log.gz 2024-09-18T00:42:41.088 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.127840.log.gz 2024-09-18T00:42:41.088 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70583.log.gz 2024-09-18T00:42:41.088 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129974.log.gz 2024-09-18T00:42:41.088 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.71294.log.gz 2024-09-18T00:42:41.088 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47144.log.gz 2024-09-18T00:42:41.088 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89088.log.gz 2024-09-18T00:42:41.088 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.0% 90.1% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.146533.log.gz 2024-09-18T00:42:41.088 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60296.log.gz 2024-09-18T00:42:41.089 INFO:teuthology.orchestra.run.smithi192.stderr: 80.0% 79.9% 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58707.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.141107.log.gz 2024-09-18T00:42:41.089 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.089 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53349.log.gz 2024-09-18T00:42:41.089 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60107.log.gz 2024-09-18T00:42:41.089 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.130877.log.gz 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.119868.log.gz 2024-09-18T00:42:41.089 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.089 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.107028.log.gz 2024-09-18T00:42:41.089 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.119107.log.gz 2024-09-18T00:42:41.089 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.134530.log.gz 2024-09-18T00:42:41.089 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.48179.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.48045.log.gz 2024-09-18T00:42:41.089 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.089 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40853.log.gz 2024-09-18T00:42:41.089 INFO:teuthology.orchestra.run.smithi192.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.94769.log.gz 2024-09-18T00:42:41.090 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87910.log.gz 2024-09-18T00:42:41.090 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.147361.log.gz 2024-09-18T00:42:41.090 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.60212.log.gz 2024-09-18T00:42:41.090 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.76148.log.gz 2024-09-18T00:42:41.090 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.48436.log.gz 2024-09-18T00:42:41.090 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.71426.log.gz 2024-09-18T00:42:41.090 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.39404.log.gz 2024-09-18T00:42:41.090 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.84412.log.gz 2024-09-18T00:42:41.090 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.148146.log.gz 2024-09-18T00:42:41.090 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.35437.log.gz 2024-09-18T00:42:41.090 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.142886.log.gz 2024-09-18T00:42:41.090 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88119.log.gz 2024-09-18T00:42:41.090 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.77576.log.gz 2024-09-18T00:42:41.091 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94795.log.gz 2024-09-18T00:42:41.091 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.140328.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.69269.log.gz 2024-09-18T00:42:41.091 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.091 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.36185.log.gz 2024-09-18T00:42:41.091 INFO:teuthology.orchestra.run.smithi192.stderr: 54.9% -- replaced with /var/log/ceph/cluster1-client.mirror.35266.log.gz 2024-09-18T00:42:41.091 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78835.log.gz 2024-09-18T00:42:41.091 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.112698.log.gz 2024-09-18T00:42:41.091 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% 87.7% 89.5% -- replaced with /var/log/ceph/cluster2-client.admin.139936.log.gz 2024-09-18T00:42:41.091 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.84014.log.gz 2024-09-18T00:42:41.091 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.104180.log.gz 2024-09-18T00:42:41.091 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.117793.log.gz 2024-09-18T00:42:41.091 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.118800.log.gz 2024-09-18T00:42:41.091 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.100818.log.gz 2024-09-18T00:42:41.092 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.106957.log.gz 2024-09-18T00:42:41.092 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.130088.log.gz 2024-09-18T00:42:41.092 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.118555.log.gz 2024-09-18T00:42:41.092 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113031.log.gz 2024-09-18T00:42:41.092 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85327.log.gz 2024-09-18T00:42:41.092 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.81353.log.gz 2024-09-18T00:42:41.092 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41065.log.gz 2024-09-18T00:42:41.092 INFO:teuthology.orchestra.run.smithi192.stderr: 19.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.35629.log.gz 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.57012.log.gz 2024-09-18T00:42:41.092 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.092 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.118668.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.76694.log.gz 2024-09-18T00:42:41.092 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.092 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47601.log.gz 2024-09-18T00:42:41.092 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.139695.log.gz 2024-09-18T00:42:41.093 INFO:teuthology.orchestra.run.smithi192.stderr: 66.1% 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.140326.log.gz 2024-09-18T00:42:41.093 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.105950.log.gz 2024-09-18T00:42:41.093 INFO:teuthology.orchestra.run.smithi192.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47279.log.gz 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.80823.log.gz 2024-09-18T00:42:41.093 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.093 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48081.log.gz 2024-09-18T00:42:41.093 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.120469.log.gz 2024-09-18T00:42:41.093 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.102978.log.gz 2024-09-18T00:42:41.093 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55406.log.gz 2024-09-18T00:42:41.093 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.118823.log.gz 2024-09-18T00:42:41.093 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.147893.log.gz 2024-09-18T00:42:41.093 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.105722.log.gz 2024-09-18T00:42:41.093 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.104213.log.gz 2024-09-18T00:42:41.093 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.139354.log.gz 2024-09-18T00:42:41.093 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.81841.log.gz 2024-09-18T00:42:41.094 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.47992.log.gz 2024-09-18T00:42:41.094 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.149156.log.gz 2024-09-18T00:42:41.094 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.81700.log.gz 2024-09-18T00:42:41.094 INFO:teuthology.orchestra.run.smithi192.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72216.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.149846.log.gz 2024-09-18T00:42:41.094 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.094 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58686.log.gz 2024-09-18T00:42:41.094 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35815.log.gz 2024-09-18T00:42:41.094 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.110762.log.gz 2024-09-18T00:42:41.094 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94131.log.gz 2024-09-18T00:42:41.094 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.47347.log.gz 2024-09-18T00:42:41.094 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.127842.log.gz 2024-09-18T00:42:41.094 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64089.log.gz 2024-09-18T00:42:41.094 INFO:teuthology.orchestra.run.smithi192.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-client.admin.116753.log.gz 2024-09-18T00:42:41.095 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.129492.log.gz 2024-09-18T00:42:41.095 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108025.log.gz 2024-09-18T00:42:41.095 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.120251.log.gz 2024-09-18T00:42:41.095 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88900.log.gz 2024-09-18T00:42:41.095 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132877.log.gz 2024-09-18T00:42:41.095 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.92356.log.gz 2024-09-18T00:42:41.095 INFO:teuthology.orchestra.run.smithi192.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.45335.log.gz 2024-09-18T00:42:41.095 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.127538.log.gz 2024-09-18T00:42:41.095 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.142492.log.gz 2024-09-18T00:42:41.095 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81981.log.gz 2024-09-18T00:42:41.095 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.130453.log.gz 2024-09-18T00:42:41.095 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70717.log.gz 2024-09-18T00:42:41.095 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.76631.log.gz 2024-09-18T00:42:41.096 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.151169.log.gz 2024-09-18T00:42:41.096 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83896.log.gz 2024-09-18T00:42:41.096 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.46977.log.gz 2024-09-18T00:42:41.096 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.107026.log.gz 90.4% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146088.log.gz 2024-09-18T00:42:41.096 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.55138.log.gz 2024-09-18T00:42:41.096 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.128137.log.gz 2024-09-18T00:42:41.096 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.115599.log.gz 2024-09-18T00:42:41.096 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89319.log.gz 2024-09-18T00:42:41.096 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.096 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41232.log.gz 2024-09-18T00:42:41.096 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.128132.log.gz 2024-09-18T00:42:41.096 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.44003.log.gz 2024-09-18T00:42:41.097 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.68084.log.gz 2024-09-18T00:42:41.097 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.129310.log.gz 2024-09-18T00:42:41.097 INFO:teuthology.orchestra.run.smithi192.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.129746.log.gz 2024-09-18T00:42:41.097 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98318.log.gz 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.129103.log.gz 2024-09-18T00:42:41.097 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42952.log.gz 2024-09-18T00:42:41.097 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.097 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.106593.log.gz 2024-09-18T00:42:41.097 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.93830.log.gz 2024-09-18T00:42:41.097 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55135.log.gz 2024-09-18T00:42:41.097 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145777.log.gz 2024-09-18T00:42:41.097 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.146299.log.gz 2024-09-18T00:42:41.097 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.123249.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.68988.log.gz 2024-09-18T00:42:41.097 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.097 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.56345.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.42702.log.gz 2024-09-18T00:42:41.098 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.098 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.129445.log.gz 2024-09-18T00:42:41.098 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.80251.log.gz 2024-09-18T00:42:41.098 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.140127.log.gz 2024-09-18T00:42:41.098 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.88582.log.gz 87.2% 2024-09-18T00:42:41.098 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.129579.log.gz 2024-09-18T00:42:41.098 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51711.log.gz 2024-09-18T00:42:41.098 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.73858.log.gz 2024-09-18T00:42:41.098 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.95688.log.gz 2024-09-18T00:42:41.098 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.122593.log.gz 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.45170.log.gz 2024-09-18T00:42:41.098 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% 2024-09-18T00:42:41.098 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.131369.log.gz 2024-09-18T00:42:41.098 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.146066.log.gz 2024-09-18T00:42:41.099 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.84192.log.gz 2024-09-18T00:42:41.099 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.70215.log.gz 2024-09-18T00:42:41.099 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119003.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.92676.log.gz 2024-09-18T00:42:41.099 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111562.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.77681.log.gz 2024-09-18T00:42:41.099 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.099 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.099 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% 87.7% 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71910.log.gz 2024-09-18T00:42:41.099 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.116380.log.gz 2024-09-18T00:42:41.099 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.71558.log.gz 2024-09-18T00:42:41.099 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.45450.log.gz 2024-09-18T00:42:41.099 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.68353.log.gz 2024-09-18T00:42:41.099 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.137321.log.gz 2024-09-18T00:42:41.099 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100333.log.gz 2024-09-18T00:42:41.100 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.151304.log.gz 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.103893.log.gz 2024-09-18T00:42:41.100 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.100 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.141905.log.gz 2024-09-18T00:42:41.100 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.75824.log.gz 2024-09-18T00:42:41.100 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36381.log.gz 2024-09-18T00:42:41.100 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.143146.log.gz 2024-09-18T00:42:41.100 INFO:teuthology.orchestra.run.smithi192.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.106712.log.gz 2024-09-18T00:42:41.100 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.91815.log.gz 2024-09-18T00:42:41.100 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.37034.log.gz 2024-09-18T00:42:41.100 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138873.log.gz 2024-09-18T00:42:41.100 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.49026.log.gz 2024-09-18T00:42:41.100 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.58955.log.gz 2024-09-18T00:42:41.100 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.145776.log.gz 2024-09-18T00:42:41.101 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.142798.log.gz 2024-09-18T00:42:41.101 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.48564.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.118114.log.gz 2024-09-18T00:42:41.101 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.101 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.127131.log.gz 2024-09-18T00:42:41.101 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.145692.log.gz 2024-09-18T00:42:41.101 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.48742.log.gz 2024-09-18T00:42:41.101 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70282.log.gz 2024-09-18T00:42:41.101 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.139873.log.gz 90.0% 2024-09-18T00:42:41.101 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.145983.log.gz 2024-09-18T00:42:41.101 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.143039.log.gz 2024-09-18T00:42:41.101 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123812.log.gz 2024-09-18T00:42:41.101 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42862.log.gz 87.3% 2024-09-18T00:42:41.101 INFO:teuthology.orchestra.run.smithi192.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98946.log.gz 2024-09-18T00:42:41.102 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.141864.log.gz 2024-09-18T00:42:41.102 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.96015.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.138477.log.gz 2024-09-18T00:42:41.102 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.102 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.86310.log.gz 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.105541.log.gz 2024-09-18T00:42:41.102 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.148694.log.gz 2024-09-18T00:42:41.102 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.102 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.93248.log.gz 2024-09-18T00:42:41.102 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.150751.log.gz 2024-09-18T00:42:41.102 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.92295.log.gz 2024-09-18T00:42:41.102 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.151693.log.gz 2024-09-18T00:42:41.102 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.151622.log.gz 2024-09-18T00:42:41.102 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.69960.log.gz 2024-09-18T00:42:41.102 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.104021.log.gz 2024-09-18T00:42:41.103 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.83672.log.gz 2024-09-18T00:42:41.103 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.151172.log.gz 2024-09-18T00:42:41.103 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.94722.log.gz 2024-09-18T00:42:41.103 INFO:teuthology.orchestra.run.smithi192.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94977.log.gz 2024-09-18T00:42:41.103 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.139042.log.gz 2024-09-18T00:42:41.103 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57588.log.gz 2024-09-18T00:42:41.103 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.131091.log.gz 2024-09-18T00:42:41.103 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.93477.log.gz 2024-09-18T00:42:41.103 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.46010.log.gz 2024-09-18T00:42:41.103 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.130911.log.gz 2024-09-18T00:42:41.103 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.116438.log.gz 2024-09-18T00:42:41.103 INFO:teuthology.orchestra.run.smithi192.stderr: 66.3% -- replaced with /var/log/ceph/cluster1-client.mirror.94608.log.gz 2024-09-18T00:42:41.103 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.80975.log.gz 2024-09-18T00:42:41.104 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.95862.log.gz 2024-09-18T00:42:41.104 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94681.log.gz 2024-09-18T00:42:41.104 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.48715.log.gz 2024-09-18T00:42:41.104 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.131523.log.gz 2024-09-18T00:42:41.104 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.87911.log.gz 2024-09-18T00:42:41.104 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118436.log.gz 2024-09-18T00:42:41.104 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.83928.log.gz 2024-09-18T00:42:41.104 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.116227.log.gz 2024-09-18T00:42:41.104 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.57306.log.gz 2024-09-18T00:42:41.104 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.80821.log.gz 2024-09-18T00:42:41.104 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.128695.log.gz 2024-09-18T00:42:41.104 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.44712.log.gz 2024-09-18T00:42:41.104 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.68330.log.gz 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.93092.log.gz 2024-09-18T00:42:41.105 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.105 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.45324.log.gz 2024-09-18T00:42:41.105 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.119216.log.gz 2024-09-18T00:42:41.105 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70852.log.gz 2024-09-18T00:42:41.105 INFO:teuthology.orchestra.run.smithi192.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.96140.log.gz 2024-09-18T00:42:41.105 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88899.log.gz 2024-09-18T00:42:41.105 INFO:teuthology.orchestra.run.smithi192.stderr: 66.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106634.log.gz 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.127219.log.gz 2024-09-18T00:42:41.105 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.105 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.118269.log.gz 2024-09-18T00:42:41.105 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59539.log.gz 2024-09-18T00:42:41.105 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.139040.log.gz 2024-09-18T00:42:41.105 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59733.log.gz 2024-09-18T00:42:41.105 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.116997.log.gz 2024-09-18T00:42:41.105 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.86142.log.gz 87.6% 2024-09-18T00:42:41.106 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.45993.log.gz 2024-09-18T00:42:41.106 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.45992.log.gz 2024-09-18T00:42:41.106 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.44703.log.gz 2024-09-18T00:42:41.106 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.104641.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.95794.log.gz 2024-09-18T00:42:41.106 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.106 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.92006.log.gz 2024-09-18T00:42:41.106 INFO:teuthology.orchestra.run.smithi192.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.107464.log.gz 2024-09-18T00:42:41.106 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.59623.log.gz 2024-09-18T00:42:41.106 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.72168.log.gz 2024-09-18T00:42:41.106 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.57269.log.gz 2024-09-18T00:42:41.106 INFO:teuthology.orchestra.run.smithi192.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.118410.log.gz 2024-09-18T00:42:41.106 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.45752.log.gz 2024-09-18T00:42:41.106 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.151358.log.gz 2024-09-18T00:42:41.107 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117524.log.gz 2024-09-18T00:42:41.107 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.45079.log.gz 2024-09-18T00:42:41.107 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.105742.log.gz 2024-09-18T00:42:41.107 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.56904.log.gz 2024-09-18T00:42:41.107 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.104640.log.gz 2024-09-18T00:42:41.107 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.45258.log.gz 2024-09-18T00:42:41.107 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.47904.log.gz 2024-09-18T00:42:41.107 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.92577.log.gz 2024-09-18T00:42:41.107 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.63750.log.gz 2024-09-18T00:42:41.107 INFO:teuthology.orchestra.run.smithi192.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129331.log.gz 2024-09-18T00:42:41.107 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.93476.log.gz 2024-09-18T00:42:41.107 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.116993.log.gz 2024-09-18T00:42:41.107 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.58727.log.gz 2024-09-18T00:42:41.108 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.116093.log.gz 2024-09-18T00:42:41.108 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.128506.log.gz 2024-09-18T00:42:41.108 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.71973.log.gz 2024-09-18T00:42:41.108 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.118041.log.gz 2024-09-18T00:42:41.108 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.130062.log.gz 2024-09-18T00:42:41.108 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.105204.log.gz 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.118533.log.gz 2024-09-18T00:42:41.108 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.108 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.48873.log.gz 2024-09-18T00:42:41.108 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.80448.log.gz 2024-09-18T00:42:41.108 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.59491.log.gz 2024-09-18T00:42:41.108 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.95061.log.gz 2024-09-18T00:42:41.108 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.131499.log.gz 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.118932.log.gz 2024-09-18T00:42:41.108 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.109 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.115603.log.gz 2024-09-18T00:42:41.109 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112232.log.gz 2024-09-18T00:42:41.109 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.94111.log.gz 2024-09-18T00:42:41.109 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.69071.log.gz 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.150842.log.gz 2024-09-18T00:42:41.109 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141408.log.gz 2024-09-18T00:42:41.109 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.109 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.143333.log.gz 2024-09-18T00:42:41.109 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.107090.log.gz 2024-09-18T00:42:41.109 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36319.log.gz 2024-09-18T00:42:41.109 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.72492.log.gz 2024-09-18T00:42:41.109 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% 90.0% 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.46002.log.gz 2024-09-18T00:42:41.109 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.47206.log.gz 2024-09-18T00:42:41.109 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.152398.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.147473.log.gz 2024-09-18T00:42:41.110 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.110 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.131043.log.gz 2024-09-18T00:42:41.110 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.148318.log.gz 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.36271.log.gz 2024-09-18T00:42:41.110 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.110 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.115543.log.gz 90.4% 2024-09-18T00:42:41.110 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.52931.log.gz 2024-09-18T00:42:41.110 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.119612.log.gz 2024-09-18T00:42:41.110 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.104935.log.gz 80.0% 2024-09-18T00:42:41.110 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.107923.log.gz 2024-09-18T00:42:41.110 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.151574.log.gz 2024-09-18T00:42:41.110 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.68143.log.gz 2024-09-18T00:42:41.110 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71039.log.gz 2024-09-18T00:42:41.110 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.69149.log.gz 2024-09-18T00:42:41.111 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.104635.log.gz 2024-09-18T00:42:41.111 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.120087.log.gz 2024-09-18T00:42:41.111 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.59069.log.gz 2024-09-18T00:42:41.111 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.129123.log.gz 2024-09-18T00:42:41.111 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.70416.log.gz 80.0% 2024-09-18T00:42:41.111 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.83978.log.gz 2024-09-18T00:42:41.111 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.68708.log.gz 2024-09-18T00:42:41.111 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.151866.log.gz 2024-09-18T00:42:41.111 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.107875.log.gz 2024-09-18T00:42:41.111 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.127731.log.gz 2024-09-18T00:42:41.111 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60566.log.gz 2024-09-18T00:42:41.111 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.139030.log.gz 2024-09-18T00:42:41.111 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.117000.log.gz 2024-09-18T00:42:41.111 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117638.log.gz 2024-09-18T00:42:41.112 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.92005.log.gz 2024-09-18T00:42:41.112 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.56687.log.gz 2024-09-18T00:42:41.112 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.95925.log.gz 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.151614.log.gz 2024-09-18T00:42:41.112 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.112 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.80367.log.gz 2024-09-18T00:42:41.112 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.148422.log.gz 2024-09-18T00:42:41.112 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.70127.log.gz 2024-09-18T00:42:41.112 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.68352.log.gz 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.44709.log.gz 2024-09-18T00:42:41.112 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.116111.log.gz 2024-09-18T00:42:41.112 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.152039.log.gz 2024-09-18T00:42:41.112 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54298.log.gz 2024-09-18T00:42:41.112 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.93789.log.gz 2024-09-18T00:42:41.112 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.113 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46687.log.gz 2024-09-18T00:42:41.113 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130584.log.gz 2024-09-18T00:42:41.113 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.70074.log.gz 2024-09-18T00:42:41.113 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58324.log.gz 2024-09-18T00:42:41.113 INFO:teuthology.orchestra.run.smithi192.stderr: 56.7% -- replaced with /var/log/ceph/cluster1-client.mirror.35334.log.gz 2024-09-18T00:42:41.113 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48394.log.gz 2024-09-18T00:42:41.113 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95313.log.gz 2024-09-18T00:42:41.113 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.93944.log.gz 2024-09-18T00:42:41.113 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.140322.log.gz 2024-09-18T00:42:41.113 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.107569.log.gz 2024-09-18T00:42:41.113 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.65033.log.gz 2024-09-18T00:42:41.113 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.104854.log.gz 90.0% 2024-09-18T00:42:41.113 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.148061.log.gz 2024-09-18T00:42:41.114 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60017.log.gz 2024-09-18T00:42:41.114 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.119218.log.gz 2024-09-18T00:42:41.114 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70054.log.gz 2024-09-18T00:42:41.114 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.129176.log.gz 2024-09-18T00:42:41.114 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.119393.log.gz 2024-09-18T00:42:41.114 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.119717.log.gz 2024-09-18T00:42:41.114 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.106800.log.gz 2024-09-18T00:42:41.114 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.71158.log.gz 2024-09-18T00:42:41.114 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36687.log.gz 2024-09-18T00:42:41.114 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster2-client.admin.140316.log.gz 2024-09-18T00:42:41.114 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.68144.log.gz 2024-09-18T00:42:41.114 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.104030.log.gz 2024-09-18T00:42:41.114 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.74624.log.gz 2024-09-18T00:42:41.114 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% 89.8% -- replaced with /var/log/ceph/cluster1-client.admin.82647.log.gz 2024-09-18T00:42:41.115 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69205.log.gz 2024-09-18T00:42:41.115 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.92064.log.gz 2024-09-18T00:42:41.115 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.107500.log.gz 2024-09-18T00:42:41.115 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.68141.log.gz 2024-09-18T00:42:41.115 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.93716.log.gz 2024-09-18T00:42:41.115 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82323.log.gz 2024-09-18T00:42:41.115 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.148503.log.gz 2024-09-18T00:42:41.115 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.147352.log.gz 2024-09-18T00:42:41.115 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.139934.log.gz 2024-09-18T00:42:41.115 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.129465.log.gz 2024-09-18T00:42:41.115 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.46306.log.gz 2024-09-18T00:42:41.115 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.128840.log.gz 2024-09-18T00:42:41.115 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46916.log.gz 2024-09-18T00:42:41.116 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130843.log.gz 2024-09-18T00:42:41.116 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90432.log.gz 2024-09-18T00:42:41.116 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.152404.log.gz 2024-09-18T00:42:41.116 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.141355.log.gz 2024-09-18T00:42:41.116 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.69003.log.gz 2024-09-18T00:42:41.116 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.admin.70899.log.gz 2024-09-18T00:42:41.116 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.68927.log.gz 2024-09-18T00:42:41.116 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.57914.log.gz 2024-09-18T00:42:41.116 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.82042.log.gz 2024-09-18T00:42:41.116 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.104182.log.gz 2024-09-18T00:42:41.116 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.36139.log.gz 2024-09-18T00:42:41.116 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.150856.log.gz 2024-09-18T00:42:41.116 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.146928.log.gz 2024-09-18T00:42:41.117 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71510.log.gz 2024-09-18T00:42:41.117 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59601.log.gz 2024-09-18T00:42:41.117 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.144850.log.gz 2024-09-18T00:42:41.117 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70925.log.gz 2024-09-18T00:42:41.117 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.138928.log.gz 2024-09-18T00:42:41.117 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.79825.log.gz 2024-09-18T00:42:41.117 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63541.log.gz 2024-09-18T00:42:41.117 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.58271.log.gz 2024-09-18T00:42:41.117 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.96224.log.gz 2024-09-18T00:42:41.117 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.94400.log.gz 2024-09-18T00:42:41.117 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.143121.log.gz 2024-09-18T00:42:41.117 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.128561.log.gz 2024-09-18T00:42:41.117 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.118322.log.gz 2024-09-18T00:42:41.118 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137661.log.gz 2024-09-18T00:42:41.118 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140216.log.gz 2024-09-18T00:42:41.118 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.83708.log.gz 2024-09-18T00:42:41.118 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51249.log.gz 2024-09-18T00:42:41.118 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.70013.log.gz 2024-09-18T00:42:41.118 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114121.log.gz 2024-09-18T00:42:41.118 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.80318.log.gz 2024-09-18T00:42:41.118 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.86228.log.gz 2024-09-18T00:42:41.118 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.46008.log.gz 90.1% 2024-09-18T00:42:41.118 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.147686.log.gz 2024-09-18T00:42:41.118 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.68332.log.gz 2024-09-18T00:42:41.118 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.150208.log.gz 2024-09-18T00:42:41.118 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.141983.log.gz 2024-09-18T00:42:41.118 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35951.log.gz 2024-09-18T00:42:41.119 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.45695.log.gz 2024-09-18T00:42:41.119 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.140323.log.gz 2024-09-18T00:42:41.119 INFO:teuthology.orchestra.run.smithi192.stderr: 66.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82830.log.gz 2024-09-18T00:42:41.119 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57174.log.gz 2024-09-18T00:42:41.119 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.admin.129948.log.gz 2024-09-18T00:42:41.119 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92756.log.gz 2024-09-18T00:42:41.119 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.91564.log.gz 2024-09-18T00:42:41.119 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.70986.log.gz 2024-09-18T00:42:41.119 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.127808.log.gz 2024-09-18T00:42:41.119 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.147536.log.gz 2024-09-18T00:42:41.119 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.86754.log.gz 2024-09-18T00:42:41.119 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57478.log.gz 2024-09-18T00:42:41.119 INFO:teuthology.orchestra.run.smithi192.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59866.log.gz 2024-09-18T00:42:41.120 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.140899.log.gz 2024-09-18T00:42:41.120 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129880.log.gz 2024-09-18T00:42:41.120 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140266.log.gz 2024-09-18T00:42:41.120 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.admin.118182.log.gz 2024-09-18T00:42:41.120 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.56340.log.gz 2024-09-18T00:42:41.120 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121463.log.gz 2024-09-18T00:42:41.120 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134635.log.gz 2024-09-18T00:42:41.120 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141522.log.gz 2024-09-18T00:42:41.120 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57550.log.gz 2024-09-18T00:42:41.120 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.36575.log.gz 2024-09-18T00:42:41.120 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141294.log.gz 2024-09-18T00:42:41.120 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54692.log.gz 2024-09-18T00:42:41.120 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.116549.log.gz 2024-09-18T00:42:41.120 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93236.log.gz 2024-09-18T00:42:41.121 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123836.log.gz 2024-09-18T00:42:41.121 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47373.log.gz 2024-09-18T00:42:41.121 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.45299.log.gz 2024-09-18T00:42:41.121 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116820.log.gz 2024-09-18T00:42:41.121 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.141469.log.gz 2024-09-18T00:42:41.121 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.139707.log.gz 2024-09-18T00:42:41.121 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.57639.log.gz 2024-09-18T00:42:41.121 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.49141.log.gz 2024-09-18T00:42:41.121 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.71116.log.gz 2024-09-18T00:42:41.121 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.92241.log.gz 2024-09-18T00:42:41.121 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.147978.log.gz 2024-09-18T00:42:41.121 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.admin.94541.log.gz 2024-09-18T00:42:41.121 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.117927.log.gz 2024-09-18T00:42:41.122 INFO:teuthology.orchestra.run.smithi192.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72405.log.gz 2024-09-18T00:42:41.122 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145543.log.gz 2024-09-18T00:42:41.122 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89590.log.gz 2024-09-18T00:42:41.122 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.82156.log.gz 2024-09-18T00:42:41.122 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57595.log.gz 2024-09-18T00:42:41.122 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.105970.log.gz 2024-09-18T00:42:41.122 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.140162.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.115465.log.gz 2024-09-18T00:42:41.122 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.122 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.147584.log.gz 2024-09-18T00:42:41.122 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.57650.log.gz 2024-09-18T00:42:41.122 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.105856.log.gz 2024-09-18T00:42:41.122 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129062.log.gz 2024-09-18T00:42:41.122 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.150838.log.gz 2024-09-18T00:42:41.122 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.109921.log.gz 2024-09-18T00:42:41.123 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.151002.log.gz 2024-09-18T00:42:41.123 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42115.log.gz 2024-09-18T00:42:41.123 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88541.log.gz 2024-09-18T00:42:41.123 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.94286.log.gz 2024-09-18T00:42:41.123 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.admin.47118.log.gz 2024-09-18T00:42:41.123 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.91977.log.gz 2024-09-18T00:42:41.123 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.70302.log.gz 2024-09-18T00:42:41.123 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.93256.log.gz 2024-09-18T00:42:41.123 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.36355.log.gz 2024-09-18T00:42:41.123 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.116228.log.gz 2024-09-18T00:42:41.123 INFO:teuthology.orchestra.run.smithi192.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.141838.log.gz 2024-09-18T00:42:41.123 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116922.log.gz 2024-09-18T00:42:41.123 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100077.log.gz 2024-09-18T00:42:41.123 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.59236.log.gz 2024-09-18T00:42:41.124 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.68585.log.gz 2024-09-18T00:42:41.124 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.140321.log.gz 2024-09-18T00:42:41.124 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.124720.log.gz 2024-09-18T00:42:41.124 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65349.log.gz 2024-09-18T00:42:41.124 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.129860.log.gz 2024-09-18T00:42:41.124 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121745.log.gz 2024-09-18T00:42:41.124 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149573.log.gz 2024-09-18T00:42:41.125 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78670.log.gz 2024-09-18T00:42:41.125 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.49584.log.gz 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.108204.log.gz 2024-09-18T00:42:41.125 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.125 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.82866.log.gz 2024-09-18T00:42:41.125 INFO:teuthology.orchestra.run.smithi192.stderr: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35590.log.gz 2024-09-18T00:42:41.125 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.139633.log.gz 2024-09-18T00:42:41.125 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.69650.log.gz 2024-09-18T00:42:41.125 INFO:teuthology.orchestra.run.smithi192.stderr: 57.7% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135073.log.gz 2024-09-18T00:42:41.125 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35539.log.gz 2024-09-18T00:42:41.125 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.110801.log.gz 2024-09-18T00:42:41.125 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.145903.log.gz 87.6% 2024-09-18T00:42:41.125 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41652.log.gz 2024-09-18T00:42:41.125 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.81351.log.gz 2024-09-18T00:42:41.126 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56144.log.gz 2024-09-18T00:42:41.126 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.56635.log.gz 2024-09-18T00:42:41.126 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.142360.log.gz 2024-09-18T00:42:41.126 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.139742.log.gz 2024-09-18T00:42:41.126 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.82551.log.gz 2024-09-18T00:42:41.126 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90603.log.gz 2024-09-18T00:42:41.126 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.102642.log.gz 2024-09-18T00:42:41.126 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71794.log.gz 2024-09-18T00:42:41.126 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62700.log.gz 2024-09-18T00:42:41.126 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.116548.log.gz 2024-09-18T00:42:41.126 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.admin.106111.log.gz 2024-09-18T00:42:41.126 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103426.log.gz 2024-09-18T00:42:41.126 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109882.log.gz 2024-09-18T00:42:41.127 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.144851.log.gz 2024-09-18T00:42:41.127 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.55238.log.gz 2024-09-18T00:42:41.127 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.146324.log.gz 2024-09-18T00:42:41.127 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54485.log.gz 2024-09-18T00:42:41.127 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74918.log.gz 2024-09-18T00:42:41.127 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.118907.log.gz 2024-09-18T00:42:41.127 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.131127.log.gz 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149875.log.gz 2024-09-18T00:42:41.127 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.127 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78037.log.gz 2024-09-18T00:42:41.127 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50227.log.gz 2024-09-18T00:42:41.127 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97289.log.gz 2024-09-18T00:42:41.127 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.138079.log.gz 2024-09-18T00:42:41.127 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.145272.log.gz 2024-09-18T00:42:41.127 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.139730.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.120046.log.gz 2024-09-18T00:42:41.128 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.128 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137952.log.gz 2024-09-18T00:42:41.128 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.107806.log.gz 2024-09-18T00:42:41.128 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71536.log.gz 2024-09-18T00:42:41.128 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.73300.log.gz 2024-09-18T00:42:41.128 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.58841.log.gz 2024-09-18T00:42:41.128 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.146847.log.gz 2024-09-18T00:42:41.128 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90347.log.gz 2024-09-18T00:42:41.128 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45461.log.gz 2024-09-18T00:42:41.128 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36927.log.gz 2024-09-18T00:42:41.128 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.68628.log.gz 2024-09-18T00:42:41.128 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.151846.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.48698.log.gz 2024-09-18T00:42:41.128 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134781.log.gz 2024-09-18T00:42:41.129 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.142071.log.gz 2024-09-18T00:42:41.129 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.129 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52785.log.gz 2024-09-18T00:42:41.129 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54860.log.gz 2024-09-18T00:42:41.129 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.106670.log.gz 2024-09-18T00:42:41.129 INFO:teuthology.orchestra.run.smithi192.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.47678.log.gz 2024-09-18T00:42:41.129 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137451.log.gz 2024-09-18T00:42:41.129 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.79154.log.gz 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49341.log.gz 2024-09-18T00:42:41.129 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.129 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39859.log.gz 2024-09-18T00:42:41.129 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41274.log.gz 2024-09-18T00:42:41.129 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.68674.log.gz 2024-09-18T00:42:41.130 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.79901.log.gz 2024-09-18T00:42:41.130 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.79090.log.gz 2024-09-18T00:42:41.130 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91565.log.gz 2024-09-18T00:42:41.130 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.75760.log.gz 2024-09-18T00:42:41.130 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48826.log.gz 2024-09-18T00:42:41.130 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89383.log.gz 2024-09-18T00:42:41.130 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78836.log.gz 2024-09-18T00:42:41.130 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.85976.log.gz 2024-09-18T00:42:41.130 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.104482.log.gz 2024-09-18T00:42:41.130 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101302.log.gz 2024-09-18T00:42:41.130 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100651.log.gz 2024-09-18T00:42:41.130 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50577.log.gz 2024-09-18T00:42:41.130 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101300.log.gz 2024-09-18T00:42:41.131 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99746.log.gz 2024-09-18T00:42:41.131 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36507.log.gz 2024-09-18T00:42:41.131 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46573.log.gz 2024-09-18T00:42:41.131 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.126888.log.gz 2024-09-18T00:42:41.131 INFO:teuthology.orchestra.run.smithi192.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.47575.log.gz 2024-09-18T00:42:41.131 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.127218.log.gz 2024-09-18T00:42:41.131 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128579.log.gz 2024-09-18T00:42:41.131 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43752.log.gz 2024-09-18T00:42:41.131 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60627.log.gz 2024-09-18T00:42:41.131 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.admin.47461.log.gz 2024-09-18T00:42:41.131 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.86889.log.gz 2024-09-18T00:42:41.131 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.45259.log.gz 2024-09-18T00:42:41.131 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.115807.log.gz 2024-09-18T00:42:41.131 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.142924.log.gz 2024-09-18T00:42:41.132 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.52635.log.gz 2024-09-18T00:42:41.132 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.admin.141610.log.gz 2024-09-18T00:42:41.132 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.80590.log.gz 2024-09-18T00:42:41.132 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.129518.log.gz 2024-09-18T00:42:41.132 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59443.log.gz 2024-09-18T00:42:41.132 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.87555.log.gz 2024-09-18T00:42:41.132 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.69649.log.gz 2024-09-18T00:42:41.132 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.124718.log.gz 2024-09-18T00:42:41.132 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76150.log.gz 2024-09-18T00:42:41.132 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.96906.log.gz 2024-09-18T00:42:41.132 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44571.log.gz 2024-09-18T00:42:41.132 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111982.log.gz 2024-09-18T00:42:41.132 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.146383.log.gz 2024-09-18T00:42:41.133 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.45090.log.gz 2024-09-18T00:42:41.133 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101802.log.gz 2024-09-18T00:42:41.133 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86643.log.gz 2024-09-18T00:42:41.133 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.103757.log.gz 2024-09-18T00:42:41.133 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40812.log.gz 2024-09-18T00:42:41.133 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.97919.log.gz 2024-09-18T00:42:41.133 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.128135.log.gz 2024-09-18T00:42:41.133 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115874.log.gz 2024-09-18T00:42:41.133 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.117471.log.gz 2024-09-18T00:42:41.133 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91061.log.gz 2024-09-18T00:42:41.133 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.47903.log.gz 2024-09-18T00:42:41.133 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134171.log.gz 2024-09-18T00:42:41.134 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.133644.log.gz 2024-09-18T00:42:41.134 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.54398.log.gz 2024-09-18T00:42:41.134 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75632.log.gz 2024-09-18T00:42:41.134 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.5% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.151728.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.106365.log.gz 2024-09-18T00:42:41.134 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.134 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.78146.log.gz 2024-09-18T00:42:41.134 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114376.log.gz 2024-09-18T00:42:41.134 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99579.log.gz 2024-09-18T00:42:41.134 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.85737.log.gz 2024-09-18T00:42:41.134 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111012.log.gz 2024-09-18T00:42:41.134 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133480.log.gz 2024-09-18T00:42:41.134 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99097.log.gz 2024-09-18T00:42:41.134 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63708.log.gz 2024-09-18T00:42:41.135 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66926.log.gz 2024-09-18T00:42:41.135 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.118871.log.gz 2024-09-18T00:42:41.135 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70966.log.gz 2024-09-18T00:42:41.135 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.134721.log.gz 2024-09-18T00:42:41.135 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.admin.106339.log.gz 2024-09-18T00:42:41.135 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36812.log.gz 2024-09-18T00:42:41.135 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.139035.log.gz 2024-09-18T00:42:41.135 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.92689.log.gz 2024-09-18T00:42:41.135 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136544.log.gz 2024-09-18T00:42:41.135 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133812.log.gz 2024-09-18T00:42:41.135 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54296.log.gz 2024-09-18T00:42:41.135 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.78312.log.gz 2024-09-18T00:42:41.135 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.113871.log.gz 2024-09-18T00:42:41.136 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134426.log.gz 2024-09-18T00:42:41.136 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72567.log.gz 2024-09-18T00:42:41.136 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.admin.141724.log.gz 2024-09-18T00:42:41.136 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.119802.log.gz 2024-09-18T00:42:41.136 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126756.log.gz 2024-09-18T00:42:41.136 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133203.log.gz 2024-09-18T00:42:41.136 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64425.log.gz 2024-09-18T00:42:41.136 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.76963.log.gz 2024-09-18T00:42:41.136 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86455.log.gz 2024-09-18T00:42:41.136 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.65935.log.gz 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87460.log.gz 2024-09-18T00:42:41.136 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.136 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.89014.log.gz 2024-09-18T00:42:41.136 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63878.log.gz 2024-09-18T00:42:41.137 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.133164.log.gz 2024-09-18T00:42:41.137 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42745.log.gz 2024-09-18T00:42:41.137 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77410.log.gz 2024-09-18T00:42:41.137 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100039.log.gz 2024-09-18T00:42:41.137 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.145226.log.gz 2024-09-18T00:42:41.137 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43333.log.gz 2024-09-18T00:42:41.137 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53918.log.gz 2024-09-18T00:42:41.137 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.84641.log.gz 2024-09-18T00:42:41.137 INFO:teuthology.orchestra.run.smithi192.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-client.admin.106225.log.gz 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90281.log.gz 2024-09-18T00:42:41.137 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126418.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.111647.log.gz 2024-09-18T00:42:41.137 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.137 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.137 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69049.log.gz 2024-09-18T00:42:41.137 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125450.log.gz 2024-09-18T00:42:41.138 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.122513.log.gz 2024-09-18T00:42:41.138 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.92195.log.gz 2024-09-18T00:42:41.138 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44005.log.gz 2024-09-18T00:42:41.138 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.134170.log.gz 2024-09-18T00:42:41.138 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123476.log.gz 2024-09-18T00:42:41.138 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41484.log.gz 2024-09-18T00:42:41.138 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75550.log.gz 2024-09-18T00:42:41.138 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.125619.log.gz 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109755.log.gz 2024-09-18T00:42:41.138 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.138 INFO:teuthology.orchestra.run.smithi192.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.36728.log.gz 2024-09-18T00:42:41.138 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148002.log.gz 2024-09-18T00:42:41.138 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.147582.log.gz 2024-09-18T00:42:41.138 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.79005.log.gz 2024-09-18T00:42:41.139 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88959.log.gz 2024-09-18T00:42:41.139 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.135139.log.gz 2024-09-18T00:42:41.139 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.80822.log.gz 2024-09-18T00:42:41.139 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54818.log.gz 2024-09-18T00:42:41.139 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88616.log.gz 2024-09-18T00:42:41.139 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.74879.log.gz 2024-09-18T00:42:41.139 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53096.log.gz 2024-09-18T00:42:41.139 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110382.log.gz 2024-09-18T00:42:41.139 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41862.log.gz 2024-09-18T00:42:41.139 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137659.log.gz 2024-09-18T00:42:41.139 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111431.log.gz 2024-09-18T00:42:41.139 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.74500.log.gz 2024-09-18T00:42:41.139 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37995.log.gz 2024-09-18T00:42:41.139 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136900.log.gz 2024-09-18T00:42:41.140 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61511.log.gz 2024-09-18T00:42:41.140 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88918.log.gz 2024-09-18T00:42:41.140 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.83129.log.gz 2024-09-18T00:42:41.140 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63120.log.gz 2024-09-18T00:42:41.140 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72252.log.gz 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128103.log.gz 2024-09-18T00:42:41.140 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.68995.log.gz 2024-09-18T00:42:41.140 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.140 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42956.log.gz 2024-09-18T00:42:41.140 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64635.log.gz 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.96530.log.gz 2024-09-18T00:42:41.140 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.140 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.45041.log.gz 2024-09-18T00:42:41.140 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41372.log.gz 2024-09-18T00:42:41.140 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54946.log.gz 2024-09-18T00:42:41.141 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.118513.log.gz 2024-09-18T00:42:41.141 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118737.log.gz 2024-09-18T00:42:41.141 INFO:teuthology.orchestra.run.smithi192.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.35505.log.gz 2024-09-18T00:42:41.141 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.121627.log.gz 90.0% 2024-09-18T00:42:41.141 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67553.log.gz 2024-09-18T00:42:41.141 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.65960.log.gz 2024-09-18T00:42:41.141 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49362.log.gz 89.9% 2024-09-18T00:42:41.141 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.65200.log.gz 2024-09-18T00:42:41.141 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.116610.log.gz 2024-09-18T00:42:41.141 INFO:teuthology.orchestra.run.smithi192.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71341.log.gz 2024-09-18T00:42:41.141 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82959.log.gz 2024-09-18T00:42:41.141 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124947.log.gz 2024-09-18T00:42:41.142 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75761.log.gz 2024-09-18T00:42:41.142 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51522.log.gz 2024-09-18T00:42:41.142 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.86352.log.gz 90.8% 2024-09-18T00:42:41.142 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.101217.log.gz 2024-09-18T00:42:41.142 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.126819.log.gz 2024-09-18T00:42:41.142 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.99850.log.gz 2024-09-18T00:42:41.142 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41653.log.gz 2024-09-18T00:42:41.142 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41022.log.gz 2024-09-18T00:42:41.142 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.151718.log.gz 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.142203.log.gz 2024-09-18T00:42:41.142 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.142 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75823.log.gz 2024-09-18T00:42:41.142 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.88875.log.gz 2024-09-18T00:42:41.142 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99156.log.gz 2024-09-18T00:42:41.142 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.138264.log.gz 2024-09-18T00:42:41.143 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62471.log.gz 2024-09-18T00:42:41.143 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36223.log.gz 2024-09-18T00:42:41.143 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67364.log.gz 2024-09-18T00:42:41.143 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36045.log.gz 2024-09-18T00:42:41.143 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114377.log.gz 2024-09-18T00:42:41.143 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.112065.log.gz 2024-09-18T00:42:41.143 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.47720.log.gz 2024-09-18T00:42:41.143 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77575.log.gz 2024-09-18T00:42:41.143 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.90512.log.gz 2024-09-18T00:42:41.143 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109435.log.gz 2024-09-18T00:42:41.143 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.117699.log.gz 2024-09-18T00:42:41.143 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36774.log.gz 2024-09-18T00:42:41.143 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.56688.log.gz 2024-09-18T00:42:41.144 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.99912.log.gz 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53685.log.gz 2024-09-18T00:42:41.144 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.144 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121736.log.gz 2024-09-18T00:42:41.144 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.97856.log.gz 2024-09-18T00:42:41.144 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51374.log.gz 2024-09-18T00:42:41.144 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76990.log.gz 2024-09-18T00:42:41.144 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124568.log.gz 2024-09-18T00:42:41.144 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.74733.log.gz 2024-09-18T00:42:41.144 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98677.log.gz 2024-09-18T00:42:41.144 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96453.log.gz 2024-09-18T00:42:41.144 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.142429.log.gz 2024-09-18T00:42:41.144 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.39005.log.gz 2024-09-18T00:42:41.144 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75088.log.gz 2024-09-18T00:42:41.144 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.94892.log.gz 2024-09-18T00:42:41.145 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.136019.log.gz 2024-09-18T00:42:41.145 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117772.log.gz 2024-09-18T00:42:41.145 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52006.log.gz 2024-09-18T00:42:41.145 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.63458.log.gz 2024-09-18T00:42:41.145 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113241.log.gz 87.3% 2024-09-18T00:42:41.145 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.129921.log.gz 2024-09-18T00:42:41.145 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114501.log.gz 2024-09-18T00:42:41.145 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44640.log.gz 2024-09-18T00:42:41.145 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.admin.105026.log.gz 2024-09-18T00:42:41.145 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98987.log.gz 2024-09-18T00:42:41.145 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125096.log.gz 2024-09-18T00:42:41.145 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71340.log.gz 2024-09-18T00:42:41.145 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75719.log.gz 2024-09-18T00:42:41.146 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109965.log.gz 2024-09-18T00:42:41.146 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79725.log.gz 2024-09-18T00:42:41.146 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82095.log.gz 2024-09-18T00:42:41.146 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64549.log.gz 2024-09-18T00:42:41.146 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88313.log.gz 2024-09-18T00:42:41.146 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100963.log.gz 2024-09-18T00:42:41.146 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.107064.log.gz 2024-09-18T00:42:41.146 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136774.log.gz 2024-09-18T00:42:41.146 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51101.log.gz 2024-09-18T00:42:41.146 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.151890.log.gz 2024-09-18T00:42:41.146 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42703.log.gz 2024-09-18T00:42:41.146 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50785.log.gz 2024-09-18T00:42:41.146 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114587.log.gz 2024-09-18T00:42:41.147 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55366.log.gz 2024-09-18T00:42:41.147 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135347.log.gz 2024-09-18T00:42:41.147 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67406.log.gz 2024-09-18T00:42:41.147 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37143.log.gz 2024-09-18T00:42:41.147 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36993.log.gz 2024-09-18T00:42:41.147 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.145155.log.gz 2024-09-18T00:42:41.147 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77534.log.gz 2024-09-18T00:42:41.147 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.69288.log.gz 2024-09-18T00:42:41.147 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121966.log.gz 2024-09-18T00:42:41.147 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35317.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.43585.log.gz 2024-09-18T00:42:41.147 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.147 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.95860.log.gz 2024-09-18T00:42:41.147 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111811.log.gz 2024-09-18T00:42:41.147 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50827.log.gz 2024-09-18T00:42:41.148 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.119458.log.gz 2024-09-18T00:42:41.148 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.149264.log.gz 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.95597.log.gz 2024-09-18T00:42:41.148 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60849.log.gz 2024-09-18T00:42:41.148 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.148 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.87954.log.gz 2024-09-18T00:42:41.148 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109501.log.gz 2024-09-18T00:42:41.148 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48941.log.gz 2024-09-18T00:42:41.148 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53413.log.gz 2024-09-18T00:42:41.148 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52593.log.gz 2024-09-18T00:42:41.148 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.99957.log.gz 2024-09-18T00:42:41.148 INFO:teuthology.orchestra.run.smithi192.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.118383.log.gz 2024-09-18T00:42:41.148 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.49114.log.gz 2024-09-18T00:42:41.148 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75574.log.gz 2024-09-18T00:42:41.149 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100753.log.gz 2024-09-18T00:42:41.149 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89951.log.gz 2024-09-18T00:42:41.149 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.150165.log.gz 2024-09-18T00:42:41.149 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.145482.log.gz 2024-09-18T00:42:41.149 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105090.log.gz 2024-09-18T00:42:41.149 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67940.log.gz 2024-09-18T00:42:41.149 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.90137.log.gz 2024-09-18T00:42:41.149 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59669.log.gz 2024-09-18T00:42:41.149 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.104425.log.gz 2024-09-18T00:42:41.149 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122175.log.gz 2024-09-18T00:42:41.149 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.149472.log.gz 2024-09-18T00:42:41.149 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121964.log.gz 2024-09-18T00:42:41.149 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.77475.log.gz 2024-09-18T00:42:41.149 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.72449.log.gz 2024-09-18T00:42:41.150 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98047.log.gz 2024-09-18T00:42:41.150 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.142644.log.gz 2024-09-18T00:42:41.150 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101427.log.gz 2024-09-18T00:42:41.150 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.69407.log.gz 2024-09-18T00:42:41.150 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49469.log.gz 2024-09-18T00:42:41.150 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66337.log.gz 2024-09-18T00:42:41.150 INFO:teuthology.orchestra.run.smithi192.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.82716.log.gz 2024-09-18T00:42:41.150 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126250.log.gz 2024-09-18T00:42:41.150 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63944.log.gz 2024-09-18T00:42:41.150 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99493.log.gz 2024-09-18T00:42:41.150 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.110552.log.gz 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53139.log.gz 2024-09-18T00:42:41.150 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.150 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133163.log.gz 2024-09-18T00:42:41.151 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113071.log.gz 2024-09-18T00:42:41.151 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44369.log.gz 2024-09-18T00:42:41.151 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50514.log.gz 2024-09-18T00:42:41.151 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124318.log.gz 2024-09-18T00:42:41.151 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50829.log.gz 2024-09-18T00:42:41.151 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75763.log.gz 2024-09-18T00:42:41.151 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.43582.log.gz 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.99788.log.gz 2024-09-18T00:42:41.151 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.151 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121568.log.gz 2024-09-18T00:42:41.151 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72671.log.gz 2024-09-18T00:42:41.151 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112484.log.gz 2024-09-18T00:42:41.151 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42326.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.51499.log.gz 2024-09-18T00:42:41.151 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.152 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.55449.log.gz 2024-09-18T00:42:41.152 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115912.log.gz 2024-09-18T00:42:41.152 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45851.log.gz 2024-09-18T00:42:41.152 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74078.log.gz 2024-09-18T00:42:41.152 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53412.log.gz 2024-09-18T00:42:41.152 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55659.log.gz 2024-09-18T00:42:41.152 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89089.log.gz 2024-09-18T00:42:41.152 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.119827.log.gz 2024-09-18T00:42:41.152 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76632.log.gz 2024-09-18T00:42:41.152 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98736.log.gz 2024-09-18T00:42:41.152 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98420.log.gz 2024-09-18T00:42:41.152 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.127874.log.gz 2024-09-18T00:42:41.153 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93769.log.gz 2024-09-18T00:42:41.153 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113702.log.gz 2024-09-18T00:42:41.153 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122219.log.gz 2024-09-18T00:42:41.153 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.133753.log.gz 2024-09-18T00:42:41.153 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36990.log.gz 2024-09-18T00:42:41.153 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51878.log.gz 2024-09-18T00:42:41.153 INFO:teuthology.orchestra.run.smithi192.stderr: 66.1% -- replaced with /var/log/ceph/cluster1-client.mirror.93924.log.gz 2024-09-18T00:42:41.153 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90851.log.gz 2024-09-18T00:42:41.153 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53223.log.gz 2024-09-18T00:42:41.153 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88815.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.50575.log.gz 2024-09-18T00:42:41.153 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.153 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41375.log.gz 2024-09-18T00:42:41.153 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41482.log.gz 2024-09-18T00:42:41.154 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.108851.log.gz 2024-09-18T00:42:41.154 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124190.log.gz 2024-09-18T00:42:41.154 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.79900.log.gz 2024-09-18T00:42:41.154 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.101298.log.gz 89.4% 2024-09-18T00:42:41.154 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.127993.log.gz 2024-09-18T00:42:41.154 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65348.log.gz 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.94085.log.gz 2024-09-18T00:42:41.154 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.68955.log.gz 2024-09-18T00:42:41.154 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.92786.log.gz 2024-09-18T00:42:41.154 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74960.log.gz 2024-09-18T00:42:41.154 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.85739.log.gz 2024-09-18T00:42:41.154 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133542.log.gz 2024-09-18T00:42:41.154 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134634.log.gz 2024-09-18T00:42:41.154 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61419.log.gz 2024-09-18T00:42:41.155 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126503.log.gz 2024-09-18T00:42:41.155 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.80792.log.gz 2024-09-18T00:42:41.155 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42022.log.gz 2024-09-18T00:42:41.155 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133368.log.gz 2024-09-18T00:42:41.155 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135011.log.gz 2024-09-18T00:42:41.155 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.155 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136648.log.gz 2024-09-18T00:42:41.155 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63877.log.gz 2024-09-18T00:42:41.155 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42423.log.gz 2024-09-18T00:42:41.155 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66270.log.gz 2024-09-18T00:42:41.155 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124987.log.gz 2024-09-18T00:42:41.155 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.76105.log.gz 2024-09-18T00:42:41.155 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.54993.log.gz 2024-09-18T00:42:41.155 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101592.log.gz 2024-09-18T00:42:41.156 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112021.log.gz 2024-09-18T00:42:41.156 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90811.log.gz 2024-09-18T00:42:41.156 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.135700.log.gz 2024-09-18T00:42:41.156 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.135012.log.gz 2024-09-18T00:42:41.156 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86454.log.gz 2024-09-18T00:42:41.156 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96249.log.gz 2024-09-18T00:42:41.156 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65808.log.gz 2024-09-18T00:42:41.156 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123016.log.gz 2024-09-18T00:42:41.156 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.139929.log.gz 2024-09-18T00:42:41.156 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98022.log.gz 2024-09-18T00:42:41.156 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122912.log.gz 2024-09-18T00:42:41.156 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% 89.9% -- replaced with /var/log/ceph/cluster2-client.admin.81056.log.gz 2024-09-18T00:42:41.156 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.98116.log.gz 2024-09-18T00:42:41.157 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.39632.log.gz 2024-09-18T00:42:41.157 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134044.log.gz 2024-09-18T00:42:41.157 INFO:teuthology.orchestra.run.smithi192.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81290.log.gz 2024-09-18T00:42:41.157 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.123015.log.gz 2024-09-18T00:42:41.157 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42955.log.gz 2024-09-18T00:42:41.157 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.100122.log.gz 2024-09-18T00:42:41.157 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.91102.log.gz 2024-09-18T00:42:41.157 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62784.log.gz 2024-09-18T00:42:41.157 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89253.log.gz 2024-09-18T00:42:41.157 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54020.log.gz 2024-09-18T00:42:41.157 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63246.log.gz 2024-09-18T00:42:41.157 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115157.log.gz 2024-09-18T00:42:41.157 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.54147.log.gz 2024-09-18T00:42:41.158 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.106064.log.gz 2024-09-18T00:42:41.158 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.82996.log.gz 2024-09-18T00:42:41.158 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40219.log.gz 2024-09-18T00:42:41.158 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50617.log.gz 2024-09-18T00:42:41.158 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.81143.log.gz 2024-09-18T00:42:41.158 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85347.log.gz 2024-09-18T00:42:41.158 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112192.log.gz 2024-09-18T00:42:41.158 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115031.log.gz 2024-09-18T00:42:41.158 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.70530.log.gz 2024-09-18T00:42:41.158 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.90853.log.gz 2024-09-18T00:42:41.158 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51375.log.gz 90.0% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41062.log.gz 2024-09-18T00:42:41.158 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.146636.log.gz 2024-09-18T00:42:41.158 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.159 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64028.log.gz 2024-09-18T00:42:41.159 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.92833.log.gz 2024-09-18T00:42:41.159 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53809.log.gz 2024-09-18T00:42:41.159 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.89506.log.gz 2024-09-18T00:42:41.159 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.139576.log.gz 2024-09-18T00:42:41.159 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63459.log.gz 90.1% 2024-09-18T00:42:41.159 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.62472.log.gz 2024-09-18T00:42:41.159 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78732.log.gz 2024-09-18T00:42:41.159 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87869.log.gz 2024-09-18T00:42:41.159 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134510.log.gz 2024-09-18T00:42:41.159 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.43798.log.gz 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.82908.log.gz 2024-09-18T00:42:41.159 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.159 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.133515.log.gz 2024-09-18T00:42:41.160 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74794.log.gz 2024-09-18T00:42:41.160 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60449.log.gz 2024-09-18T00:42:41.160 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75464.log.gz 2024-09-18T00:42:41.160 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149113.log.gz 2024-09-18T00:42:41.160 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114588.log.gz 2024-09-18T00:42:41.160 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112277.log.gz 2024-09-18T00:42:41.160 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125703.log.gz 2024-09-18T00:42:41.160 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98606.log.gz 2024-09-18T00:42:41.160 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.57918.log.gz 2024-09-18T00:42:41.160 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.131300.log.gz 2024-09-18T00:42:41.160 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.admin.129834.log.gz 2024-09-18T00:42:41.160 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.145123.log.gz 2024-09-18T00:42:41.160 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.65201.log.gz 2024-09-18T00:42:41.161 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78460.log.gz 2024-09-18T00:42:41.161 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77995.log.gz 2024-09-18T00:42:41.161 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.92618.log.gz 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105101.log.gz 2024-09-18T00:42:41.161 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.9% 2024-09-18T00:42:41.161 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.97686.log.gz 2024-09-18T00:42:41.161 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.36469.log.gz 2024-09-18T00:42:41.161 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137741.log.gz 2024-09-18T00:42:41.161 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.119648.log.gz 2024-09-18T00:42:41.161 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39731.log.gz 2024-09-18T00:42:41.161 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71647.log.gz 2024-09-18T00:42:41.161 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.122702.log.gz 2024-09-18T00:42:41.161 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116928.log.gz 2024-09-18T00:42:41.161 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39000.log.gz 2024-09-18T00:42:41.162 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.129807.log.gz 2024-09-18T00:42:41.162 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59333.log.gz 2024-09-18T00:42:41.162 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102560.log.gz 2024-09-18T00:42:41.162 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.135199.log.gz 2024-09-18T00:42:41.162 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93099.log.gz 2024-09-18T00:42:41.162 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62239.log.gz 2024-09-18T00:42:41.162 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99327.log.gz 2024-09-18T00:42:41.162 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.126855.log.gz 2024-09-18T00:42:41.162 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.136061.log.gz 2024-09-18T00:42:41.162 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74734.log.gz 2024-09-18T00:42:41.162 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.73282.log.gz 2024-09-18T00:42:41.162 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42534.log.gz 2024-09-18T00:42:41.162 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.89799.log.gz 2024-09-18T00:42:41.163 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125179.log.gz 2024-09-18T00:42:41.163 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.139039.log.gz 2024-09-18T00:42:41.163 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.45257.log.gz 2024-09-18T00:42:41.163 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39149.log.gz 2024-09-18T00:42:41.163 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.53962.log.gz 2024-09-18T00:42:41.163 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61836.log.gz 2024-09-18T00:42:41.163 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126607.log.gz 2024-09-18T00:42:41.163 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76760.log.gz 2024-09-18T00:42:41.163 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59122.log.gz 2024-09-18T00:42:41.163 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67595.log.gz 2024-09-18T00:42:41.163 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.97314.log.gz 2024-09-18T00:42:41.163 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65264.log.gz 2024-09-18T00:42:41.163 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.105929.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.55617.log.gz 2024-09-18T00:42:41.164 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.164 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88960.log.gz 2024-09-18T00:42:41.164 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86374.log.gz 2024-09-18T00:42:41.164 INFO:teuthology.orchestra.run.smithi192.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.76340.log.gz 2024-09-18T00:42:41.164 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.142033.log.gz 2024-09-18T00:42:41.164 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137576.log.gz 2024-09-18T00:42:41.164 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.145357.log.gz 2024-09-18T00:42:41.164 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.49159.log.gz 2024-09-18T00:42:41.164 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41182.log.gz 2024-09-18T00:42:41.164 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65177.log.gz 2024-09-18T00:42:41.164 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121505.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.91950.log.gz 2024-09-18T00:42:41.164 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.164 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53099.log.gz 2024-09-18T00:42:41.165 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38577.log.gz 2024-09-18T00:42:41.165 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.95383.log.gz 2024-09-18T00:42:41.165 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40854.log.gz 2024-09-18T00:42:41.165 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148737.log.gz 2024-09-18T00:42:41.165 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55701.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.37207.log.gz 2024-09-18T00:42:41.165 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.165 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63666.log.gz 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86644.log.gz 2024-09-18T00:42:41.165 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.165 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.66338.log.gz 2024-09-18T00:42:41.165 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.102433.log.gz 2024-09-18T00:42:41.165 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.96519.log.gz 2024-09-18T00:42:41.165 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136215.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.136545.log.gz 2024-09-18T00:42:41.165 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.165 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.110931.log.gz 2024-09-18T00:42:41.166 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.139970.log.gz 2024-09-18T00:42:41.166 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.107158.log.gz 2024-09-18T00:42:41.166 INFO:teuthology.orchestra.run.smithi192.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.118363.log.gz 2024-09-18T00:42:41.166 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.135685.log.gz 2024-09-18T00:42:41.166 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43282.log.gz 2024-09-18T00:42:41.166 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98275.log.gz 2024-09-18T00:42:41.166 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.113744.log.gz 2024-09-18T00:42:41.166 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.100167.log.gz 2024-09-18T00:42:41.166 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42954.log.gz 2024-09-18T00:42:41.166 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98445.log.gz 2024-09-18T00:42:41.166 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.69414.log.gz 2024-09-18T00:42:41.166 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.0% 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.114378.log.gz 2024-09-18T00:42:41.167 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.101299.log.gz 2024-09-18T00:42:41.167 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.149028.log.gz 2024-09-18T00:42:41.167 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41906.log.gz 2024-09-18T00:42:41.167 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.77620.log.gz 2024-09-18T00:42:41.167 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125744.log.gz 2024-09-18T00:42:41.167 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.96473.log.gz 2024-09-18T00:42:41.167 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44511.log.gz 2024-09-18T00:42:41.167 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60913.log.gz 2024-09-18T00:42:41.167 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.87699.log.gz 2024-09-18T00:42:41.167 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.83298.log.gz 2024-09-18T00:42:41.167 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124696.log.gz 2024-09-18T00:42:41.168 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136374.log.gz 2024-09-18T00:42:41.168 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64548.log.gz 2024-09-18T00:42:41.168 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.136125.log.gz 2024-09-18T00:42:41.168 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.80500.log.gz 2024-09-18T00:42:41.168 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76464.log.gz 2024-09-18T00:42:41.168 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142297.log.gz 2024-09-18T00:42:41.168 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.137950.log.gz 2024-09-18T00:42:41.168 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67488.log.gz 2024-09-18T00:42:41.168 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113958.log.gz 2024-09-18T00:42:41.168 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90557.log.gz 2024-09-18T00:42:41.168 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.115066.log.gz 89.9% 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.65830.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.36047.log.gz 2024-09-18T00:42:41.168 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.168 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54990.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.57648.log.gz 2024-09-18T00:42:41.169 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.169 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.169 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66105.log.gz 2024-09-18T00:42:41.169 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44004.log.gz 2024-09-18T00:42:41.169 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112861.log.gz 2024-09-18T00:42:41.169 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.36088.log.gz 2024-09-18T00:42:41.169 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41139.log.gz 2024-09-18T00:42:41.169 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112908.log.gz 2024-09-18T00:42:41.169 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.87761.log.gz 2024-09-18T00:42:41.169 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60105.log.gz 90.1% 2024-09-18T00:42:41.169 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.40009.log.gz 2024-09-18T00:42:41.169 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.150185.log.gz 2024-09-18T00:42:41.169 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112278.log.gz 2024-09-18T00:42:41.170 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.38136.log.gz 2024-09-18T00:42:41.170 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.149450.log.gz 2024-09-18T00:42:41.170 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37364.log.gz 2024-09-18T00:42:41.170 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.92928.log.gz 2024-09-18T00:42:41.170 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121420.log.gz 2024-09-18T00:42:41.170 INFO:teuthology.orchestra.run.smithi192.stderr: 66.1% 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83825.log.gz 2024-09-18T00:42:41.170 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54482.log.gz 2024-09-18T00:42:41.170 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53290.log.gz 2024-09-18T00:42:41.170 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.117907.log.gz 90.0% 2024-09-18T00:42:41.170 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89062.log.gz 2024-09-18T00:42:41.170 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.145353.log.gz 2024-09-18T00:42:41.170 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.110596.log.gz 2024-09-18T00:42:41.170 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149157.log.gz 2024-09-18T00:42:41.170 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89550.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.48943.log.gz 2024-09-18T00:42:41.171 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.171 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.108703.log.gz 2024-09-18T00:42:41.171 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136398.log.gz 2024-09-18T00:42:41.171 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.80735.log.gz 2024-09-18T00:42:41.171 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.89003.log.gz 2024-09-18T00:42:41.171 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74373.log.gz 2024-09-18T00:42:41.171 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.58754.log.gz 2024-09-18T00:42:41.171 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112191.log.gz 2024-09-18T00:42:41.171 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72607.log.gz 2024-09-18T00:42:41.171 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51331.log.gz 2024-09-18T00:42:41.171 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72061.log.gz 89.9% 2024-09-18T00:42:41.171 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.64026.log.gz 2024-09-18T00:42:41.171 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144218.log.gz 2024-09-18T00:42:41.172 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102561.log.gz 2024-09-18T00:42:41.172 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.110512.log.gz 2024-09-18T00:42:41.172 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.77265.log.gz 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73887.log.gz 2024-09-18T00:42:41.172 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.172 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.84039.log.gz 2024-09-18T00:42:41.172 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90616.log.gz 2024-09-18T00:42:41.172 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41904.log.gz 2024-09-18T00:42:41.172 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48241.log.gz 2024-09-18T00:42:41.172 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137530.log.gz 2024-09-18T00:42:41.172 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111017.log.gz 2024-09-18T00:42:41.172 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126041.log.gz 2024-09-18T00:42:41.172 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.131597.log.gz 2024-09-18T00:42:41.172 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103024.log.gz 2024-09-18T00:42:41.173 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.80626.log.gz 2024-09-18T00:42:41.173 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.65539.log.gz 2024-09-18T00:42:41.173 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.139249.log.gz 2024-09-18T00:42:41.173 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.104932.log.gz 2024-09-18T00:42:41.173 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.88542.log.gz 2024-09-18T00:42:41.173 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.85861.log.gz 2024-09-18T00:42:41.173 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102772.log.gz 2024-09-18T00:42:41.173 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113662.log.gz 2024-09-18T00:42:41.173 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37998.log.gz 2024-09-18T00:42:41.173 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.86585.log.gz 2024-09-18T00:42:41.173 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52256.log.gz 2024-09-18T00:42:41.173 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123688.log.gz 2024-09-18T00:42:41.173 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122554.log.gz 2024-09-18T00:42:41.174 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137238.log.gz 2024-09-18T00:42:41.174 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.119388.log.gz 2024-09-18T00:42:41.174 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121569.log.gz 2024-09-18T00:42:41.174 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72540.log.gz 2024-09-18T00:42:41.174 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90433.log.gz 2024-09-18T00:42:41.174 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87272.log.gz 2024-09-18T00:42:41.174 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53287.log.gz 2024-09-18T00:42:41.174 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63940.log.gz 2024-09-18T00:42:41.174 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.96847.log.gz 2024-09-18T00:42:41.174 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.126564.log.gz 2024-09-18T00:42:41.174 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40551.log.gz 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134590.log.gz 2024-09-18T00:42:41.174 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.174 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86038.log.gz 2024-09-18T00:42:41.174 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63035.log.gz 2024-09-18T00:42:41.175 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98990.log.gz 2024-09-18T00:42:41.175 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121094.log.gz 2024-09-18T00:42:41.175 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.59355.log.gz 2024-09-18T00:42:41.175 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137740.log.gz 2024-09-18T00:42:41.175 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.91141.log.gz 2024-09-18T00:42:41.175 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40855.log.gz 2024-09-18T00:42:41.175 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121734.log.gz 2024-09-18T00:42:41.175 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79301.log.gz 2024-09-18T00:42:41.175 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.90931.log.gz 2024-09-18T00:42:41.175 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112906.log.gz 2024-09-18T00:42:41.175 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111771.log.gz 2024-09-18T00:42:41.175 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62786.log.gz 2024-09-18T00:42:41.175 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87021.log.gz 2024-09-18T00:42:41.176 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52741.log.gz 2024-09-18T00:42:41.176 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113746.log.gz 2024-09-18T00:42:41.176 INFO:teuthology.orchestra.run.smithi192.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.118021.log.gz 2024-09-18T00:42:41.176 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136608.log.gz 2024-09-18T00:42:41.176 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42587.log.gz 2024-09-18T00:42:41.176 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75940.log.gz 2024-09-18T00:42:41.176 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64088.log.gz 2024-09-18T00:42:41.176 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125955.log.gz 2024-09-18T00:42:41.176 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.90011.log.gz 2024-09-18T00:42:41.176 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42112.log.gz 2024-09-18T00:42:41.176 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43793.log.gz 2024-09-18T00:42:41.176 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38729.log.gz 2024-09-18T00:42:41.176 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.99305.log.gz 2024-09-18T00:42:41.177 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.44172.log.gz 2024-09-18T00:42:41.177 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102012.log.gz 2024-09-18T00:42:41.177 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134087.log.gz 2024-09-18T00:42:41.177 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75887.log.gz 2024-09-18T00:42:41.177 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79214.log.gz 2024-09-18T00:42:41.177 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.97812.log.gz 2024-09-18T00:42:41.177 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145143.log.gz 2024-09-18T00:42:41.177 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62659.log.gz 2024-09-18T00:42:41.177 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102352.log.gz 2024-09-18T00:42:41.177 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110178.log.gz 2024-09-18T00:42:41.177 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.93231.log.gz 2024-09-18T00:42:41.177 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.72015.log.gz 2024-09-18T00:42:41.177 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.150978.log.gz 2024-09-18T00:42:41.178 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135495.log.gz 2024-09-18T00:42:41.178 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102477.log.gz 2024-09-18T00:42:41.178 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48368.log.gz 2024-09-18T00:42:41.178 INFO:teuthology.orchestra.run.smithi192.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142167.log.gz 2024-09-18T00:42:41.178 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78205.log.gz 2024-09-18T00:42:41.178 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135872.log.gz 2024-09-18T00:42:41.178 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.95795.log.gz 2024-09-18T00:42:41.178 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.86225.log.gz 2024-09-18T00:42:41.178 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126461.log.gz 2024-09-18T00:42:41.178 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39360.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.113112.log.gz 2024-09-18T00:42:41.178 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.178 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114331.log.gz 2024-09-18T00:42:41.178 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.116611.log.gz 2024-09-18T00:42:41.179 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116869.log.gz 2024-09-18T00:42:41.179 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89549.log.gz 2024-09-18T00:42:41.179 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137929.log.gz 2024-09-18T00:42:41.179 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42742.log.gz 2024-09-18T00:42:41.179 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126251.log.gz 2024-09-18T00:42:41.179 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.127323.log.gz 2024-09-18T00:42:41.179 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43374.log.gz 2024-09-18T00:42:41.179 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.76844.log.gz 2024-09-18T00:42:41.179 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123139.log.gz 2024-09-18T00:42:41.179 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.99098.log.gz 2024-09-18T00:42:41.179 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126332.log.gz 2024-09-18T00:42:41.179 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87826.log.gz 2024-09-18T00:42:41.179 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121334.log.gz 2024-09-18T00:42:41.179 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66146.log.gz 2024-09-18T00:42:41.180 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.108704.log.gz 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75254.log.gz 2024-09-18T00:42:41.180 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75298.log.gz 2024-09-18T00:42:41.180 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.180 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.112066.log.gz 2024-09-18T00:42:41.180 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63143.log.gz 2024-09-18T00:42:41.180 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66482.log.gz 2024-09-18T00:42:41.180 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.116850.log.gz 2024-09-18T00:42:41.180 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.115368.log.gz 2024-09-18T00:42:41.180 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.127536.log.gz 2024-09-18T00:42:41.180 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.68480.log.gz 2024-09-18T00:42:41.180 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.70355.log.gz 2024-09-18T00:42:41.180 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.93181.log.gz 2024-09-18T00:42:41.180 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.120816.log.gz 2024-09-18T00:42:41.181 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109752.log.gz 2024-09-18T00:42:41.181 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.88666.log.gz 2024-09-18T00:42:41.181 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102853.log.gz 2024-09-18T00:42:41.181 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130995.log.gz 2024-09-18T00:42:41.181 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101382.log.gz 2024-09-18T00:42:41.181 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.56769.log.gz 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.140838.log.gz 2024-09-18T00:42:41.181 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.181 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.144849.log.gz 2024-09-18T00:42:41.181 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101426.log.gz 2024-09-18T00:42:41.181 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.78104.log.gz 2024-09-18T00:42:41.181 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41907.log.gz 2024-09-18T00:42:41.181 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135894.log.gz 2024-09-18T00:42:41.181 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62869.log.gz 2024-09-18T00:42:41.182 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.84234.log.gz 2024-09-18T00:42:41.182 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.90488.log.gz 2024-09-18T00:42:41.182 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51439.log.gz 2024-09-18T00:42:41.182 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62955.log.gz 2024-09-18T00:42:41.182 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85002.log.gz 2024-09-18T00:42:41.182 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 80.1% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.71099.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.135767.log.gz 2024-09-18T00:42:41.182 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.182 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50474.log.gz 2024-09-18T00:42:41.182 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.77619.log.gz 2024-09-18T00:42:41.182 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36471.log.gz 2024-09-18T00:42:41.182 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114082.log.gz 2024-09-18T00:42:41.182 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% 91.0% -- replaced with /var/log/ceph/cluster1-client.admin.129720.log.gz 2024-09-18T00:42:41.182 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67449.log.gz 2024-09-18T00:42:41.182 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.71710.log.gz 2024-09-18T00:42:41.183 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66817.log.gz 2024-09-18T00:42:41.183 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.48895.log.gz 2024-09-18T00:42:41.183 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103326.log.gz 2024-09-18T00:42:41.183 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87398.log.gz 2024-09-18T00:42:41.183 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43476.log.gz 2024-09-18T00:42:41.183 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.145063.log.gz 2024-09-18T00:42:41.183 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144426.log.gz 2024-09-18T00:42:41.183 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64844.log.gz 2024-09-18T00:42:41.183 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79342.log.gz 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76191.log.gz 2024-09-18T00:42:41.183 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84454.log.gz 2024-09-18T00:42:41.183 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.183 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122410.log.gz 2024-09-18T00:42:41.183 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116883.log.gz 2024-09-18T00:42:41.183 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64506.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.63268.log.gz 2024-09-18T00:42:41.184 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.184 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102686.log.gz 2024-09-18T00:42:41.184 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138077.log.gz 2024-09-18T00:42:41.184 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124232.log.gz 2024-09-18T00:42:41.184 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122132.log.gz 2024-09-18T00:42:41.184 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90852.log.gz 2024-09-18T00:42:41.184 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113661.log.gz 2024-09-18T00:42:41.184 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.85974.log.gz 2024-09-18T00:42:41.184 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.77895.log.gz 2024-09-18T00:42:41.184 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67941.log.gz 2024-09-18T00:42:41.184 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55072.log.gz 89.9% 2024-09-18T00:42:41.184 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.86309.log.gz 2024-09-18T00:42:41.184 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.133754.log.gz 2024-09-18T00:42:41.184 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55136.log.gz 2024-09-18T00:42:41.185 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65537.log.gz 2024-09-18T00:42:41.185 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.94567.log.gz 2024-09-18T00:42:41.185 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37427.log.gz 2024-09-18T00:42:41.185 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75673.log.gz 2024-09-18T00:42:41.185 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.141583.log.gz 2024-09-18T00:42:41.185 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54650.log.gz 90.1% 2024-09-18T00:42:41.185 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54072.log.gz 2024-09-18T00:42:41.185 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.149743.log.gz 2024-09-18T00:42:41.185 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40089.log.gz 2024-09-18T00:42:41.185 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.80178.log.gz 2024-09-18T00:42:41.185 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71403.log.gz 2024-09-18T00:42:41.185 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62283.log.gz 2024-09-18T00:42:41.186 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110804.log.gz 2024-09-18T00:42:41.186 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.66734.log.gz 2024-09-18T00:42:41.186 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48788.log.gz 2024-09-18T00:42:41.186 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.88788.log.gz 2024-09-18T00:42:41.186 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.120907.log.gz 2024-09-18T00:42:41.186 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.135469.log.gz 2024-09-18T00:42:41.186 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.57140.log.gz 2024-09-18T00:42:41.186 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55616.log.gz 2024-09-18T00:42:41.186 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51037.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.43375.log.gz 2024-09-18T00:42:41.186 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.186 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98150.log.gz 2024-09-18T00:42:41.186 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90611.log.gz 2024-09-18T00:42:41.186 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.136166.log.gz 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.118606.log.gz 2024-09-18T00:42:41.187 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.187 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51187.log.gz 2024-09-18T00:42:41.187 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40639.log.gz 2024-09-18T00:42:41.187 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123416.log.gz 2024-09-18T00:42:41.187 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.90012.log.gz 2024-09-18T00:42:41.187 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85697.log.gz 2024-09-18T00:42:41.187 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.109753.log.gz 2024-09-18T00:42:41.187 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62532.log.gz 2024-09-18T00:42:41.187 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75089.log.gz 2024-09-18T00:42:41.187 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.148818.log.gz 2024-09-18T00:42:41.187 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137237.log.gz 2024-09-18T00:42:41.187 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53877.log.gz 2024-09-18T00:42:41.187 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.114375.log.gz 2024-09-18T00:42:41.187 INFO:teuthology.orchestra.run.smithi192.stderr: 89.6% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90721.log.gz 2024-09-18T00:42:41.188 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.130427.log.gz 2024-09-18T00:42:41.188 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.144725.log.gz 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44554.log.gz 2024-09-18T00:42:41.188 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.188 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110342.log.gz 2024-09-18T00:42:41.188 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52675.log.gz 2024-09-18T00:42:41.188 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56030.log.gz 2024-09-18T00:42:41.188 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.57317.log.gz 2024-09-18T00:42:41.188 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.126164.log.gz 2024-09-18T00:42:41.188 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78881.log.gz 2024-09-18T00:42:41.188 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138140.log.gz 2024-09-18T00:42:41.188 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76295.log.gz 2024-09-18T00:42:41.188 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43792.log.gz 2024-09-18T00:42:41.189 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124652.log.gz 2024-09-18T00:42:41.189 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40179.log.gz 2024-09-18T00:42:41.189 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64591.log.gz 2024-09-18T00:42:41.189 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144506.log.gz 2024-09-18T00:42:41.189 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136249.log.gz 2024-09-18T00:42:41.189 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60411.log.gz 2024-09-18T00:42:41.189 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114167.log.gz 2024-09-18T00:42:41.189 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.57325.log.gz 2024-09-18T00:42:41.189 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.74521.log.gz 2024-09-18T00:42:41.189 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134096.log.gz 2024-09-18T00:42:41.189 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.122637.log.gz 2024-09-18T00:42:41.189 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78945.log.gz 2024-09-18T00:42:41.189 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.77535.log.gz 2024-09-18T00:42:41.189 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.80721.log.gz 2024-09-18T00:42:41.190 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40937.log.gz 2024-09-18T00:42:41.190 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113115.log.gz 2024-09-18T00:42:41.190 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62995.log.gz 2024-09-18T00:42:41.190 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.91168.log.gz 2024-09-18T00:42:41.190 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138774.log.gz 89.4% 2024-09-18T00:42:41.190 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59705.log.gz 2024-09-18T00:42:41.190 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.90899.log.gz 2024-09-18T00:42:41.190 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.admin.105997.log.gz 2024-09-18T00:42:41.190 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122008.log.gz 2024-09-18T00:42:41.190 INFO:teuthology.orchestra.run.smithi192.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.admin.82297.log.gz 2024-09-18T00:42:41.190 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97287.log.gz 2024-09-18T00:42:41.190 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122914.log.gz 89.9% 2024-09-18T00:42:41.190 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.72124.log.gz 2024-09-18T00:42:41.191 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64298.log.gz 2024-09-18T00:42:41.191 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99367.log.gz 2024-09-18T00:42:41.191 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.151898.log.gz 2024-09-18T00:42:41.191 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131479.log.gz 2024-09-18T00:42:41.191 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42073.log.gz 2024-09-18T00:42:41.191 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124379.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.133395.log.gz 2024-09-18T00:42:41.191 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.191 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99159.log.gz 2024-09-18T00:42:41.191 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54440.log.gz 2024-09-18T00:42:41.191 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51078.log.gz 2024-09-18T00:42:41.191 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146172.log.gz 2024-09-18T00:42:41.191 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84653.log.gz 2024-09-18T00:42:41.191 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.77199.log.gz 2024-09-18T00:42:41.192 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63145.log.gz 2024-09-18T00:42:41.192 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.87553.log.gz 2024-09-18T00:42:41.192 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110388.log.gz 2024-09-18T00:42:41.192 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101092.log.gz 2024-09-18T00:42:41.192 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.102687.log.gz 2024-09-18T00:42:41.192 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51919.log.gz 2024-09-18T00:42:41.192 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149953.log.gz 2024-09-18T00:42:41.192 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101760.log.gz 2024-09-18T00:42:41.192 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.107679.log.gz 2024-09-18T00:42:41.192 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.136440.log.gz 90.1% 2024-09-18T00:42:41.192 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.102979.log.gz 2024-09-18T00:42:41.192 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.86519.log.gz 91.6% 2024-09-18T00:42:41.193 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.57454.log.gz 2024-09-18T00:42:41.193 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.149783.log.gz 2024-09-18T00:42:41.193 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126292.log.gz 2024-09-18T00:42:41.193 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53831.log.gz 2024-09-18T00:42:41.193 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43164.log.gz 2024-09-18T00:42:41.193 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125860.log.gz 2024-09-18T00:42:41.193 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47071.log.gz 2024-09-18T00:42:41.193 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89380.log.gz 2024-09-18T00:42:41.193 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.56098.log.gz 2024-09-18T00:42:41.193 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112691.log.gz 2024-09-18T00:42:41.193 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50450.log.gz 2024-09-18T00:42:41.193 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144505.log.gz 2024-09-18T00:42:41.193 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135534.log.gz 2024-09-18T00:42:41.194 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.91021.log.gz 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53180.log.gz 2024-09-18T00:42:41.194 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.194 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86143.log.gz 2024-09-18T00:42:41.194 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76843.log.gz 2024-09-18T00:42:41.194 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48178.log.gz 2024-09-18T00:42:41.194 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108226.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.148212.log.gz 2024-09-18T00:42:41.194 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.194 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.137111.log.gz 2024-09-18T00:42:41.194 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62827.log.gz 2024-09-18T00:42:41.194 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.150854.log.gz 2024-09-18T00:42:41.194 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60876.log.gz 2024-09-18T00:42:41.194 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.105795.log.gz 2024-09-18T00:42:41.194 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122762.log.gz 2024-09-18T00:42:41.195 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123248.log.gz 2024-09-18T00:42:41.195 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.113032.log.gz 2024-09-18T00:42:41.195 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134797.log.gz 2024-09-18T00:42:41.195 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87019.log.gz 2024-09-18T00:42:41.195 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.147053.log.gz 2024-09-18T00:42:41.195 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145585.log.gz 2024-09-18T00:42:41.195 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.62367.log.gz 2024-09-18T00:42:41.195 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.39799.log.gz 2024-09-18T00:42:41.195 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67279.log.gz 2024-09-18T00:42:41.195 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52782.log.gz 2024-09-18T00:42:41.195 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138162.log.gz 2024-09-18T00:42:41.195 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.122678.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.149238.log.gz 2024-09-18T00:42:41.195 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.195 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.65683.log.gz 2024-09-18T00:42:41.196 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.104219.log.gz 2024-09-18T00:42:41.196 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.86835.log.gz 2024-09-18T00:42:41.196 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.108180.log.gz 2024-09-18T00:42:41.196 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89273.log.gz 2024-09-18T00:42:41.196 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125025.log.gz 2024-09-18T00:42:41.196 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.147685.log.gz 2024-09-18T00:42:41.196 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148001.log.gz 2024-09-18T00:42:41.196 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78585.log.gz 2024-09-18T00:42:41.196 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103245.log.gz 2024-09-18T00:42:41.196 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.149684.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.143230.log.gz 2024-09-18T00:42:41.196 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.196 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.58526.log.gz 2024-09-18T00:42:41.196 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.149874.log.gz 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.79726.log.gz 2024-09-18T00:42:41.196 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.197 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.129378.log.gz 2024-09-18T00:42:41.197 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136270.log.gz 2024-09-18T00:42:41.197 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99703.log.gz 2024-09-18T00:42:41.197 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124510.log.gz 2024-09-18T00:42:41.197 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51437.log.gz 2024-09-18T00:42:41.197 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85821.log.gz 2024-09-18T00:42:41.197 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.92482.log.gz 2024-09-18T00:42:41.197 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.148504.log.gz 2024-09-18T00:42:41.197 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110932.log.gz 2024-09-18T00:42:41.197 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50164.log.gz 2024-09-18T00:42:41.197 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64908.log.gz 2024-09-18T00:42:41.197 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124150.log.gz 2024-09-18T00:42:41.198 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66882.log.gz 2024-09-18T00:42:41.198 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.89927.log.gz 2024-09-18T00:42:41.198 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.147894.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.136902.log.gz 2024-09-18T00:42:41.198 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.198 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.94494.log.gz 2024-09-18T00:42:41.198 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.46822.log.gz 2024-09-18T00:42:41.198 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.52299.log.gz 2024-09-18T00:42:41.198 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.123056.log.gz 2024-09-18T00:42:41.198 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124443.log.gz 2024-09-18T00:42:41.198 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.101172.log.gz 2024-09-18T00:42:41.198 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91518.log.gz 2024-09-18T00:42:41.198 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.150056.log.gz 2024-09-18T00:42:41.198 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.100846.log.gz 2024-09-18T00:42:41.199 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130165.log.gz 2024-09-18T00:42:41.199 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.admin.116766.log.gz 2024-09-18T00:42:41.199 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102139.log.gz 2024-09-18T00:42:41.199 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113951.log.gz 2024-09-18T00:42:41.199 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.89462.log.gz 2024-09-18T00:42:41.199 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87762.log.gz 2024-09-18T00:42:41.199 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43283.log.gz 2024-09-18T00:42:41.199 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121204.log.gz 2024-09-18T00:42:41.199 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113114.log.gz 2024-09-18T00:42:41.199 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53727.log.gz 2024-09-18T00:42:41.199 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.99431.log.gz 2024-09-18T00:42:41.199 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.100628.log.gz 2024-09-18T00:42:41.199 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.119764.log.gz 2024-09-18T00:42:41.199 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.147138.log.gz 2024-09-18T00:42:41.200 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.88516.log.gz 2024-09-18T00:42:41.200 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88270.log.gz 2024-09-18T00:42:41.200 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.150450.log.gz 2024-09-18T00:42:41.200 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43072.log.gz 2024-09-18T00:42:41.200 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.88249.log.gz 2024-09-18T00:42:41.200 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84436.log.gz 2024-09-18T00:42:41.200 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.70758.log.gz 2024-09-18T00:42:41.200 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40132.log.gz 2024-09-18T00:42:41.200 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63793.log.gz 2024-09-18T00:42:41.200 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76821.log.gz 2024-09-18T00:42:41.200 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.95247.log.gz 2024-09-18T00:42:41.200 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50701.log.gz 2024-09-18T00:42:41.201 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.149363.log.gz 2024-09-18T00:42:41.201 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.111225.log.gz 2024-09-18T00:42:41.201 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86376.log.gz 2024-09-18T00:42:41.201 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122092.log.gz 2024-09-18T00:42:41.201 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79596.log.gz 2024-09-18T00:42:41.201 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.133437.log.gz 2024-09-18T00:42:41.201 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114955.log.gz 2024-09-18T00:42:41.201 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.117384.log.gz 2024-09-18T00:42:41.201 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.79899.log.gz 2024-09-18T00:42:41.201 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.148633.log.gz 2024-09-18T00:42:41.201 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.143495.log.gz 2024-09-18T00:42:41.201 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75360.log.gz 2024-09-18T00:42:41.201 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.114164.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.116557.log.gz 2024-09-18T00:42:41.202 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.202 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66018.log.gz 2024-09-18T00:42:41.202 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94245.log.gz 2024-09-18T00:42:41.202 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122890.log.gz 2024-09-18T00:42:41.202 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.admin.117840.log.gz 2024-09-18T00:42:41.202 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78206.log.gz 2024-09-18T00:42:41.202 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44371.log.gz 2024-09-18T00:42:41.202 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52466.log.gz 2024-09-18T00:42:41.202 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.133227.log.gz 2024-09-18T00:42:41.202 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112696.log.gz 2024-09-18T00:42:41.202 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47487.log.gz 2024-09-18T00:42:41.202 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122763.log.gz 2024-09-18T00:42:41.202 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38578.log.gz 2024-09-18T00:42:41.202 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.86647.log.gz 2024-09-18T00:42:41.202 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.47698.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.99799.log.gz 2024-09-18T00:42:41.203 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.203 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134864.log.gz 2024-09-18T00:42:41.203 INFO:teuthology.orchestra.run.smithi192.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.118249.log.gz 2024-09-18T00:42:41.203 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.125493.log.gz 2024-09-18T00:42:41.203 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.135074.log.gz 2024-09-18T00:42:41.203 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123415.log.gz 2024-09-18T00:42:41.203 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87207.log.gz 2024-09-18T00:42:41.203 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144726.log.gz 2024-09-18T00:42:41.203 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40600.log.gz 2024-09-18T00:42:41.203 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50870.log.gz 2024-09-18T00:42:41.203 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38999.log.gz 2024-09-18T00:42:41.203 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145587.log.gz 2024-09-18T00:42:41.203 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.100650.log.gz 2024-09-18T00:42:41.203 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112862.log.gz 2024-09-18T00:42:41.204 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.98676.log.gz 2024-09-18T00:42:41.204 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85698.log.gz 2024-09-18T00:42:41.204 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125788.log.gz 2024-09-18T00:42:41.204 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137783.log.gz 2024-09-18T00:42:41.204 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.102780.log.gz 2024-09-18T00:42:41.204 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90014.log.gz 2024-09-18T00:42:41.204 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66104.log.gz 2024-09-18T00:42:41.204 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109757.log.gz 2024-09-18T00:42:41.204 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.40430.log.gz 2024-09-18T00:42:41.204 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39800.log.gz 2024-09-18T00:42:41.204 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135265.log.gz 2024-09-18T00:42:41.204 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62409.log.gz 2024-09-18T00:42:41.204 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89129.log.gz 2024-09-18T00:42:41.204 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.110971.log.gz 2024-09-18T00:42:41.205 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102056.log.gz 2024-09-18T00:42:41.205 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101512.log.gz 2024-09-18T00:42:41.205 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.54881.log.gz 91.1% -- replaced with /var/log/ceph/cluster1-client.admin.47004.log.gz 2024-09-18T00:42:41.205 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.108386.log.gz 2024-09-18T00:42:41.205 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.205 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.150102.log.gz 2024-09-18T00:42:41.205 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41695.log.gz 2024-09-18T00:42:41.205 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60938.log.gz 2024-09-18T00:42:41.205 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88584.log.gz 2024-09-18T00:42:41.205 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135389.log.gz 2024-09-18T00:42:41.205 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.54947.log.gz 2024-09-18T00:42:41.205 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123838.log.gz 2024-09-18T00:42:41.205 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.55766.log.gz 2024-09-18T00:42:41.206 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123182.log.gz 2024-09-18T00:42:41.206 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.148632.log.gz 2024-09-18T00:42:41.206 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126934.log.gz 2024-09-18T00:42:41.206 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.119259.log.gz 2024-09-18T00:42:41.206 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73344.log.gz 2024-09-18T00:42:41.206 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.95924.log.gz 2024-09-18T00:42:41.206 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110805.log.gz 2024-09-18T00:42:41.206 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114731.log.gz 2024-09-18T00:42:41.206 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.63269.log.gz 2024-09-18T00:42:41.206 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.83670.log.gz 2024-09-18T00:42:41.206 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74162.log.gz 2024-09-18T00:42:41.206 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133815.log.gz 2024-09-18T00:42:41.206 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.145774.log.gz 2024-09-18T00:42:41.206 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.98360.log.gz 2024-09-18T00:42:41.207 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50321.log.gz 2024-09-18T00:42:41.207 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 66.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129901.log.gz 2024-09-18T00:42:41.207 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.141677.log.gz 2024-09-18T00:42:41.207 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.106520.log.gz 2024-09-18T00:42:41.207 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.77955.log.gz 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47528.log.gz 2024-09-18T00:42:41.207 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.207 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-client.admin.116677.log.gz 2024-09-18T00:42:41.207 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70624.log.gz 2024-09-18T00:42:41.207 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.137844.log.gz 2024-09-18T00:42:41.207 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137953.log.gz 2024-09-18T00:42:41.207 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51755.log.gz 2024-09-18T00:42:41.207 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.104022.log.gz 2024-09-18T00:42:41.207 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51668.log.gz 2024-09-18T00:42:41.208 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65641.log.gz 2024-09-18T00:42:41.208 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39631.log.gz 2024-09-18T00:42:41.208 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38939.log.gz 2024-09-18T00:42:41.208 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63310.log.gz 2024-09-18T00:42:41.208 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.138434.log.gz 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.135556.log.gz 2024-09-18T00:42:41.208 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145062.log.gz 2024-09-18T00:42:41.208 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.208 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35798.log.gz 2024-09-18T00:42:41.208 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77365.log.gz 2024-09-18T00:42:41.208 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125262.log.gz 2024-09-18T00:42:41.208 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136754.log.gz 2024-09-18T00:42:41.208 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.87102.log.gz 2024-09-18T00:42:41.208 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98048.log.gz 2024-09-18T00:42:41.209 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39001.log.gz 2024-09-18T00:42:41.209 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50226.log.gz 2024-09-18T00:42:41.209 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112487.log.gz 2024-09-18T00:42:41.209 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 66.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82022.log.gz 2024-09-18T00:42:41.209 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.124486.log.gz 2024-09-18T00:42:41.209 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.105681.log.gz 2024-09-18T00:42:41.209 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99702.log.gz 2024-09-18T00:42:41.209 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113116.log.gz 2024-09-18T00:42:41.209 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111014.log.gz 2024-09-18T00:42:41.209 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.106023.log.gz 2024-09-18T00:42:41.209 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88583.log.gz 2024-09-18T00:42:41.209 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.138225.log.gz 2024-09-18T00:42:41.209 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54356.log.gz 2024-09-18T00:42:41.210 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114583.log.gz 2024-09-18T00:42:41.210 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111851.log.gz 2024-09-18T00:42:41.210 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.143431.log.gz 2024-09-18T00:42:41.210 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110172.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.96188.log.gz 2024-09-18T00:42:41.210 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.210 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.84302.log.gz 2024-09-18T00:42:41.210 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50533.log.gz 2024-09-18T00:42:41.210 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.145483.log.gz 2024-09-18T00:42:41.210 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.152318.log.gz 2024-09-18T00:42:41.210 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.87274.log.gz 2024-09-18T00:42:41.210 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146382.log.gz 2024-09-18T00:42:41.210 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126628.log.gz 2024-09-18T00:42:41.210 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125327.log.gz 2024-09-18T00:42:41.211 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.115258.log.gz 2024-09-18T00:42:41.211 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112401.log.gz 2024-09-18T00:42:41.211 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114922.log.gz 2024-09-18T00:42:41.211 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98906.log.gz 2024-09-18T00:42:41.211 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40279.log.gz 2024-09-18T00:42:41.211 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40321.log.gz 2024-09-18T00:42:41.211 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98780.log.gz 90.0% 2024-09-18T00:42:41.211 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.115153.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.123536.log.gz 2024-09-18T00:42:41.211 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.211 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.88157.log.gz 2024-09-18T00:42:41.211 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39004.log.gz 2024-09-18T00:42:41.211 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.144805.log.gz 2024-09-18T00:42:41.211 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97728.log.gz 2024-09-18T00:42:41.212 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102643.log.gz 2024-09-18T00:42:41.212 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64338.log.gz 2024-09-18T00:42:41.212 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108275.log.gz 2024-09-18T00:42:41.212 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.119305.log.gz 2024-09-18T00:42:41.212 INFO:teuthology.orchestra.run.smithi192.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.94058.log.gz 2024-09-18T00:42:41.212 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113701.log.gz 2024-09-18T00:42:41.212 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.47091.log.gz 2024-09-18T00:42:41.212 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65640.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.128007.log.gz 2024-09-18T00:42:41.212 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.212 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.109542.log.gz 2024-09-18T00:42:41.212 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.94172.log.gz 2024-09-18T00:42:41.212 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.65094.log.gz 2024-09-18T00:42:41.212 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37323.log.gz 2024-09-18T00:42:41.213 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72291.log.gz 2024-09-18T00:42:41.213 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.114584.log.gz 2024-09-18T00:42:41.213 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109283.log.gz 2024-09-18T00:42:41.213 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77324.log.gz 2024-09-18T00:42:41.213 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.107462.log.gz 2024-09-18T00:42:41.213 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.67575.log.gz 2024-09-18T00:42:41.213 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54568.log.gz 2024-09-18T00:42:41.213 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.55340.log.gz 2024-09-18T00:42:41.213 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43794.log.gz 2024-09-18T00:42:41.213 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123517.log.gz 2024-09-18T00:42:41.213 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51438.log.gz 2024-09-18T00:42:41.213 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.83518.log.gz 2024-09-18T00:42:41.213 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56063.log.gz 2024-09-18T00:42:41.214 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.97353.log.gz 2024-09-18T00:42:41.214 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111641.log.gz 2024-09-18T00:42:41.214 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 86.5% -- replaced with /var/log/ceph/cluster2-client.admin.139628.log.gz 2024-09-18T00:42:41.214 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101570.log.gz 2024-09-18T00:42:41.214 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.125241.log.gz 2024-09-18T00:42:41.214 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51521.log.gz 2024-09-18T00:42:41.214 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.89651.log.gz 2024-09-18T00:42:41.214 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.123643.log.gz 2024-09-18T00:42:41.214 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.133034.log.gz 2024-09-18T00:42:41.214 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88013.log.gz 2024-09-18T00:42:41.214 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71208.log.gz 2024-09-18T00:42:41.214 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110807.log.gz 2024-09-18T00:42:41.214 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.85934.log.gz 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.104957.log.gz 2024-09-18T00:42:41.215 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.215 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61919.log.gz 2024-09-18T00:42:41.215 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.143648.log.gz 2024-09-18T00:42:41.215 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66271.log.gz 2024-09-18T00:42:41.215 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61918.log.gz 2024-09-18T00:42:41.215 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.133540.log.gz 2024-09-18T00:42:41.215 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66422.log.gz 2024-09-18T00:42:41.215 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.137720.log.gz 2024-09-18T00:42:41.215 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90323.log.gz 2024-09-18T00:42:41.215 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91488.log.gz 2024-09-18T00:42:41.215 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54527.log.gz 2024-09-18T00:42:41.215 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.96518.log.gz 2024-09-18T00:42:41.215 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.98487.log.gz 2024-09-18T00:42:41.216 INFO:teuthology.orchestra.run.smithi192.stderr: 80.2% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102519.log.gz 2024-09-18T00:42:41.216 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.131264.log.gz 2024-09-18T00:42:41.216 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.139361.log.gz 2024-09-18T00:42:41.216 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130185.log.gz 2024-09-18T00:42:41.216 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65598.log.gz 2024-09-18T00:42:41.216 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53138.log.gz 2024-09-18T00:42:41.216 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.117004.log.gz 2024-09-18T00:42:41.216 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54880.log.gz 2024-09-18T00:42:41.216 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.144938.log.gz 2024-09-18T00:42:41.216 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101637.log.gz 2024-09-18T00:42:41.216 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.69400.log.gz 2024-09-18T00:42:41.216 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.150103.log.gz 2024-09-18T00:42:41.216 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73514.log.gz 2024-09-18T00:42:41.217 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135640.log.gz 2024-09-18T00:42:41.217 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51628.log.gz 2024-09-18T00:42:41.217 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41023.log.gz 2024-09-18T00:42:41.217 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.148424.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.94836.log.gz 2024-09-18T00:42:41.217 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.217 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.68709.log.gz 2024-09-18T00:42:41.217 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.121160.log.gz 2024-09-18T00:42:41.217 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78272.log.gz 2024-09-18T00:42:41.217 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110802.log.gz 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.147558.log.gz 2024-09-18T00:42:41.217 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 2024-09-18T00:42:41.217 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.90767.log.gz 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110301.log.gz 2024-09-18T00:42:41.217 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.218 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74690.log.gz 2024-09-18T00:42:41.218 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134273.log.gz 2024-09-18T00:42:41.218 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43262.log.gz 2024-09-18T00:42:41.218 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62470.log.gz 2024-09-18T00:42:41.218 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45943.log.gz 2024-09-18T00:42:41.218 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65514.log.gz 2024-09-18T00:42:41.218 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.77263.log.gz 2024-09-18T00:42:41.218 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74458.log.gz 2024-09-18T00:42:41.218 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.134361.log.gz 2024-09-18T00:42:41.218 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.117726.log.gz 2024-09-18T00:42:41.218 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.74311.log.gz 2024-09-18T00:42:41.218 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.90180.log.gz 2024-09-18T00:42:41.218 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40640.log.gz 2024-09-18T00:42:41.218 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40644.log.gz 2024-09-18T00:42:41.218 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.78462.log.gz 2024-09-18T00:42:41.219 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.88059.log.gz 2024-09-18T00:42:41.219 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63312.log.gz 2024-09-18T00:42:41.219 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.97897.log.gz 2024-09-18T00:42:41.219 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.142299.log.gz 2024-09-18T00:42:41.219 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133077.log.gz 2024-09-18T00:42:41.219 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120881.log.gz 2024-09-18T00:42:41.219 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123938.log.gz 2024-09-18T00:42:41.219 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76423.log.gz 2024-09-18T00:42:41.219 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.107699.log.gz 2024-09-18T00:42:41.219 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76715.log.gz 2024-09-18T00:42:41.219 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.145436.log.gz 2024-09-18T00:42:41.219 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121799.log.gz 2024-09-18T00:42:41.219 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114752.log.gz 2024-09-18T00:42:41.220 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.43913.log.gz 2024-09-18T00:42:41.220 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64800.log.gz 2024-09-18T00:42:41.220 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.107768.log.gz 2024-09-18T00:42:41.220 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.87953.log.gz 2024-09-18T00:42:41.220 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122258.log.gz 2024-09-18T00:42:41.220 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43584.log.gz 2024-09-18T00:42:41.220 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.85169.log.gz 2024-09-18T00:42:41.220 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.100882.log.gz 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126122.log.gz 2024-09-18T00:42:41.220 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.220 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52298.log.gz 2024-09-18T00:42:41.220 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.97857.log.gz 2024-09-18T00:42:41.220 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122636.log.gz 2024-09-18T00:42:41.220 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125828.log.gz 2024-09-18T00:42:41.221 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.84391.log.gz 2024-09-18T00:42:41.221 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56143.log.gz 2024-09-18T00:42:41.221 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.149054.log.gz 2024-09-18T00:42:41.221 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122342.log.gz 2024-09-18T00:42:41.221 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121628.log.gz 2024-09-18T00:42:41.221 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43583.log.gz 2024-09-18T00:42:41.221 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.107615.log.gz 2024-09-18T00:42:41.221 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.87617.log.gz 2024-09-18T00:42:41.221 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.143077.log.gz 2024-09-18T00:42:41.221 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52029.log.gz 2024-09-18T00:42:41.221 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109751.log.gz 2024-09-18T00:42:41.221 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.102432.log.gz 2024-09-18T00:42:41.221 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44370.log.gz 2024-09-18T00:42:41.221 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136124.log.gz 2024-09-18T00:42:41.222 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119521.log.gz 2024-09-18T00:42:41.222 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.100038.log.gz 2024-09-18T00:42:41.222 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.133310.log.gz 2024-09-18T00:42:41.222 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137384.log.gz 2024-09-18T00:42:41.222 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55787.log.gz 2024-09-18T00:42:41.222 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.112061.log.gz 2024-09-18T00:42:41.222 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43963.log.gz 2024-09-18T00:42:41.222 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122703.log.gz 2024-09-18T00:42:41.222 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.68864.log.gz 2024-09-18T00:42:41.222 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67154.log.gz 2024-09-18T00:42:41.222 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90812.log.gz 2024-09-18T00:42:41.222 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149872.log.gz 2024-09-18T00:42:41.222 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50344.log.gz 2024-09-18T00:42:41.223 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.110598.log.gz 2024-09-18T00:42:41.223 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100752.log.gz 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38665.log.gz 2024-09-18T00:42:41.223 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76004.log.gz 2024-09-18T00:42:41.223 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.223 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44002.log.gz 2024-09-18T00:42:41.223 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42233.log.gz 2024-09-18T00:42:41.223 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137871.log.gz 2024-09-18T00:42:41.223 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51795.log.gz 2024-09-18T00:42:41.223 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.142251.log.gz 2024-09-18T00:42:41.223 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.77323.log.gz 2024-09-18T00:42:41.223 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102100.log.gz 2024-09-18T00:42:41.223 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110177.log.gz 2024-09-18T00:42:41.223 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125139.log.gz 2024-09-18T00:42:41.223 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43123.log.gz 2024-09-18T00:42:41.224 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65725.log.gz 2024-09-18T00:42:41.224 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.147433.log.gz 2024-09-18T00:42:41.224 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.86730.log.gz 2024-09-18T00:42:41.224 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54567.log.gz 2024-09-18T00:42:41.224 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51753.log.gz 2024-09-18T00:42:41.224 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135808.log.gz 87.5% 2024-09-18T00:42:41.224 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.37184.log.gz 2024-09-18T00:42:41.224 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.151090.log.gz 2024-09-18T00:42:41.224 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55239.log.gz 2024-09-18T00:42:41.224 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64254.log.gz 2024-09-18T00:42:41.224 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.74668.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.42533.log.gz 2024-09-18T00:42:41.224 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.224 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.73283.log.gz 2024-09-18T00:42:41.225 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.55490.log.gz 2024-09-18T00:42:41.225 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90221.log.gz 2024-09-18T00:42:41.225 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66987.log.gz 2024-09-18T00:42:41.225 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50786.log.gz 2024-09-18T00:42:41.225 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125620.log.gz 2024-09-18T00:42:41.225 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65034.log.gz 2024-09-18T00:42:41.225 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73343.log.gz 2024-09-18T00:42:41.225 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.38135.log.gz 2024-09-18T00:42:41.225 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145366.log.gz 2024-09-18T00:42:41.225 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% 89.9% -- replaced with /var/log/ceph/cluster2-client.admin.45748.log.gz 2024-09-18T00:42:41.225 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.100649.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.141268.log.gz 2024-09-18T00:42:41.225 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.225 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65204.log.gz 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78500.log.gz 2024-09-18T00:42:41.226 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.120953.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.41442.log.gz 2024-09-18T00:42:41.226 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.226 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.226 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.107959.log.gz 2024-09-18T00:42:41.226 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.90136.log.gz 2024-09-18T00:42:41.226 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.57654.log.gz 2024-09-18T00:42:41.226 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113492.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.133604.log.gz 2024-09-18T00:42:41.226 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.226 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126375.log.gz 2024-09-18T00:42:41.226 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.125017.log.gz 2024-09-18T00:42:41.226 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137089.log.gz 2024-09-18T00:42:41.226 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.113491.log.gz 2024-09-18T00:42:41.226 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.85170.log.gz 2024-09-18T00:42:41.227 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.80503.log.gz 2024-09-18T00:42:41.227 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75253.log.gz 2024-09-18T00:42:41.227 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.135138.log.gz 2024-09-18T00:42:41.227 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98526.log.gz 2024-09-18T00:42:41.227 INFO:teuthology.orchestra.run.smithi192.stderr: 89.7% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42652.log.gz 2024-09-18T00:42:41.227 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.132838.log.gz 2024-09-18T00:42:41.227 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37324.log.gz 2024-09-18T00:42:41.227 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.83271.log.gz 2024-09-18T00:42:41.227 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52718.log.gz 2024-09-18T00:42:41.227 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67616.log.gz 2024-09-18T00:42:41.227 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.96186.log.gz 2024-09-18T00:42:41.227 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40935.log.gz 2024-09-18T00:42:41.227 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62238.log.gz 2024-09-18T00:42:41.227 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.77894.log.gz 2024-09-18T00:42:41.228 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75361.log.gz 2024-09-18T00:42:41.228 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55513.log.gz 2024-09-18T00:42:41.228 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75318.log.gz 2024-09-18T00:42:41.228 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90558.log.gz 2024-09-18T00:42:41.228 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.66815.log.gz 2024-09-18T00:42:41.228 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53767.log.gz 2024-09-18T00:42:41.228 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88060.log.gz 2024-09-18T00:42:41.228 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43542.log.gz 2024-09-18T00:42:41.228 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.94742.log.gz 2024-09-18T00:42:41.228 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121798.log.gz 2024-09-18T00:42:41.228 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68055.log.gz 2024-09-18T00:42:41.228 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.77812.log.gz 2024-09-18T00:42:41.228 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67239.log.gz 2024-09-18T00:42:41.229 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89838.log.gz 2024-09-18T00:42:41.229 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.131569.log.gz 2024-09-18T00:42:41.229 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126208.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.65956.log.gz 2024-09-18T00:42:41.229 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.229 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53484.log.gz 2024-09-18T00:42:41.229 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101636.log.gz 2024-09-18T00:42:41.229 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43165.log.gz 2024-09-18T00:42:41.229 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.141154.log.gz 2024-09-18T00:42:41.229 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.44171.log.gz 2024-09-18T00:42:41.229 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.142771.log.gz 2024-09-18T00:42:41.229 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.98567.log.gz 2024-09-18T00:42:41.229 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62913.log.gz 2024-09-18T00:42:41.229 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.145083.log.gz 2024-09-18T00:42:41.229 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148505.log.gz 2024-09-18T00:42:41.230 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.108492.log.gz 2024-09-18T00:42:41.230 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123436.log.gz 2024-09-18T00:42:41.230 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74520.log.gz 2024-09-18T00:42:41.230 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51330.log.gz 2024-09-18T00:42:41.230 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66986.log.gz 2024-09-18T00:42:41.230 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123392.log.gz 2024-09-18T00:42:41.230 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60825.log.gz 2024-09-18T00:42:41.230 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.89211.log.gz 2024-09-18T00:42:41.230 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.127079.log.gz 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61920.log.gz 2024-09-18T00:42:41.230 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.230 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.148064.log.gz 2024-09-18T00:42:41.230 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39212.log.gz 2024-09-18T00:42:41.230 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73513.log.gz 2024-09-18T00:42:41.231 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.86384.log.gz 2024-09-18T00:42:41.231 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.149954.log.gz 89.9% 2024-09-18T00:42:41.231 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.74228.log.gz 2024-09-18T00:42:41.231 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.40934.log.gz 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.145586.log.gz 2024-09-18T00:42:41.231 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133481.log.gz 2024-09-18T00:42:41.231 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97646.log.gz 2024-09-18T00:42:41.231 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.231 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128578.log.gz 2024-09-18T00:42:41.231 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91353.log.gz 2024-09-18T00:42:41.231 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.113535.log.gz 2024-09-18T00:42:41.231 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.149534.log.gz 2024-09-18T00:42:41.231 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39213.log.gz 2024-09-18T00:42:41.231 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39589.log.gz 2024-09-18T00:42:41.231 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.70697.log.gz 2024-09-18T00:42:41.232 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.82592.log.gz 2024-09-18T00:42:41.232 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36317.log.gz 2024-09-18T00:42:41.232 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53415.log.gz 2024-09-18T00:42:41.232 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.44475.log.gz 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.114122.log.gz 2024-09-18T00:42:41.232 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.232 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75982.log.gz 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65100.log.gz 2024-09-18T00:42:41.232 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.148904.log.gz 2024-09-18T00:42:41.232 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.232 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121503.log.gz 2024-09-18T00:42:41.232 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.74919.log.gz 2024-09-18T00:42:41.232 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76904.log.gz 2024-09-18T00:42:41.232 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.64675.log.gz 2024-09-18T00:42:41.233 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 66.8% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98528.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.141221.log.gz 2024-09-18T00:42:41.233 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.233 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54105.log.gz 2024-09-18T00:42:41.233 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.110595.log.gz 2024-09-18T00:42:41.233 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123582.log.gz 2024-09-18T00:42:41.233 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.135348.log.gz 2024-09-18T00:42:41.233 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.101007.log.gz 2024-09-18T00:42:41.233 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.101047.log.gz 2024-09-18T00:42:41.233 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122051.log.gz 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125534.log.gz 2024-09-18T00:42:41.233 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42536.log.gz 2024-09-18T00:42:41.233 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.233 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121838.log.gz 2024-09-18T00:42:41.233 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125009.log.gz 2024-09-18T00:42:41.233 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.103085.log.gz 2024-09-18T00:42:41.234 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.118643.log.gz 2024-09-18T00:42:41.234 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90641.log.gz 2024-09-18T00:42:41.234 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.37404.log.gz 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75633.log.gz 2024-09-18T00:42:41.234 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54695.log.gz 2024-09-18T00:42:41.234 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.234 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.98840.log.gz 2024-09-18T00:42:41.234 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.44886.log.gz 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52385.log.gz 2024-09-18T00:42:41.234 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.234 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.149658.log.gz 2024-09-18T00:42:41.234 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65430.log.gz 2024-09-18T00:42:41.234 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99660.log.gz 2024-09-18T00:42:41.234 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50269.log.gz 2024-09-18T00:42:41.234 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55658.log.gz 2024-09-18T00:42:41.235 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111855.log.gz 2024-09-18T00:42:41.235 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63751.log.gz 2024-09-18T00:42:41.235 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.145247.log.gz 2024-09-18T00:42:41.235 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122804.log.gz 2024-09-18T00:42:41.235 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53919.log.gz 2024-09-18T00:42:41.235 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72832.log.gz 2024-09-18T00:42:41.235 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120905.log.gz 2024-09-18T00:42:41.235 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65894.log.gz 2024-09-18T00:42:41.235 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53876.log.gz 2024-09-18T00:42:41.235 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99492.log.gz 2024-09-18T00:42:41.235 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121840.log.gz 2024-09-18T00:42:41.235 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.76064.log.gz 2024-09-18T00:42:41.235 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143297.log.gz 2024-09-18T00:42:41.235 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.71246.log.gz 2024-09-18T00:42:41.236 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102518.log.gz 2024-09-18T00:42:41.236 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.145879.log.gz 2024-09-18T00:42:41.236 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50869.log.gz 2024-09-18T00:42:41.236 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.85932.log.gz 2024-09-18T00:42:41.236 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90932.log.gz 2024-09-18T00:42:41.236 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121419.log.gz 2024-09-18T00:42:41.236 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102936.log.gz 2024-09-18T00:42:41.236 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64799.log.gz 2024-09-18T00:42:41.236 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.86646.log.gz 2024-09-18T00:42:41.236 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52930.log.gz 2024-09-18T00:42:41.236 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138653.log.gz 2024-09-18T00:42:41.236 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42863.log.gz 2024-09-18T00:42:41.236 INFO:teuthology.orchestra.run.smithi192.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.mirror.150669.log.gz 2024-09-18T00:42:41.236 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.117954.log.gz 2024-09-18T00:42:41.237 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146321.log.gz 2024-09-18T00:42:41.237 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.107831.log.gz 2024-09-18T00:42:41.237 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111018.log.gz 2024-09-18T00:42:41.237 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.admin.141382.log.gz 2024-09-18T00:42:41.237 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122133.log.gz 2024-09-18T00:42:41.237 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.137805.log.gz 2024-09-18T00:42:41.237 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109543.log.gz 2024-09-18T00:42:41.237 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75047.log.gz 2024-09-18T00:42:41.237 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113326.log.gz 2024-09-18T00:42:41.237 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100250.log.gz 2024-09-18T00:42:41.237 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.145856.log.gz 2024-09-18T00:42:41.237 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.39941.log.gz 2024-09-18T00:42:41.237 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134045.log.gz 2024-09-18T00:42:41.238 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.131757.log.gz 2024-09-18T00:42:41.238 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63374.log.gz 2024-09-18T00:42:41.238 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.149575.log.gz 2024-09-18T00:42:41.238 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.85003.log.gz 2024-09-18T00:42:41.238 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.119001.log.gz 2024-09-18T00:42:41.238 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.84899.log.gz 2024-09-18T00:42:41.238 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137323.log.gz 2024-09-18T00:42:41.238 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.84642.log.gz 2024-09-18T00:42:41.238 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40433.log.gz 2024-09-18T00:42:41.238 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53288.log.gz 2024-09-18T00:42:41.238 INFO:teuthology.orchestra.run.smithi192.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60143.log.gz 2024-09-18T00:42:41.238 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.84633.log.gz 2024-09-18T00:42:41.238 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136775.log.gz 2024-09-18T00:42:41.238 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68855.log.gz 2024-09-18T00:42:41.239 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.99198.log.gz 2024-09-18T00:42:41.239 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56209.log.gz 2024-09-18T00:42:41.239 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.125386.log.gz 2024-09-18T00:42:41.239 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97917.log.gz 2024-09-18T00:42:41.239 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138457.log.gz 2024-09-18T00:42:41.239 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133122.log.gz 2024-09-18T00:42:41.239 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125912.log.gz 2024-09-18T00:42:41.239 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132156.log.gz 2024-09-18T00:42:41.239 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.86891.log.gz 2024-09-18T00:42:41.239 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55070.log.gz 2024-09-18T00:42:41.239 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148903.log.gz 2024-09-18T00:42:41.239 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63833.log.gz 2024-09-18T00:42:41.239 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62366.log.gz 2024-09-18T00:42:41.239 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.134930.log.gz 2024-09-18T00:42:41.240 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.96907.log.gz 2024-09-18T00:42:41.240 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40011.log.gz 2024-09-18T00:42:41.240 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.130521.log.gz 2024-09-18T00:42:41.240 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.99282.log.gz 2024-09-18T00:42:41.240 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63079.log.gz 2024-09-18T00:42:41.240 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40280.log.gz 2024-09-18T00:42:41.240 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.78920.log.gz 2024-09-18T00:42:41.240 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86456.log.gz 2024-09-18T00:42:41.240 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123728.log.gz 2024-09-18T00:42:41.240 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.100417.log.gz 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63205.log.gz 2024-09-18T00:42:41.240 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75044.log.gz 2024-09-18T00:42:41.240 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.240 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60474.log.gz 2024-09-18T00:42:41.241 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66525.log.gz 2024-09-18T00:42:41.241 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.150483.log.gz 2024-09-18T00:42:41.241 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99890.log.gz 2024-09-18T00:42:41.241 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39861.log.gz 2024-09-18T00:42:41.241 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52551.log.gz 89.9% 2024-09-18T00:42:41.241 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42492.log.gz 2024-09-18T00:42:41.241 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42124.log.gz 2024-09-18T00:42:41.241 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43474.log.gz 2024-09-18T00:42:41.241 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.122174.log.gz 2024-09-18T00:42:41.241 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138740.log.gz 2024-09-18T00:42:41.241 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52846.log.gz 2024-09-18T00:42:41.241 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54735.log.gz 2024-09-18T00:42:41.241 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113282.log.gz 2024-09-18T00:42:41.242 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134610.log.gz 2024-09-18T00:42:41.242 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64843.log.gz 2024-09-18T00:42:41.242 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110551.log.gz 2024-09-18T00:42:41.242 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.65766.log.gz 2024-09-18T00:42:41.242 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52426.log.gz 2024-09-18T00:42:41.242 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.103724.log.gz 2024-09-18T00:42:41.242 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122973.log.gz 2024-09-18T00:42:41.242 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.48330.log.gz 2024-09-18T00:42:41.242 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.118869.log.gz 2024-09-18T00:42:41.242 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113912.log.gz 2024-09-18T00:42:41.242 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136649.log.gz 2024-09-18T00:42:41.242 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86814.log.gz 2024-09-18T00:42:41.242 INFO:teuthology.orchestra.run.smithi192.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47772.log.gz 2024-09-18T00:42:41.242 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.147791.log.gz 2024-09-18T00:42:41.243 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79045.log.gz 2024-09-18T00:42:41.243 INFO:teuthology.orchestra.run.smithi192.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.119155.log.gz 2024-09-18T00:42:41.243 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102353.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.112022.log.gz 2024-09-18T00:42:41.243 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.243 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110761.log.gz 2024-09-18T00:42:41.243 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111772.log.gz 2024-09-18T00:42:41.243 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75130.log.gz 2024-09-18T00:42:41.243 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.144784.log.gz 2024-09-18T00:42:41.243 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138891.log.gz 2024-09-18T00:42:41.243 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149574.log.gz 2024-09-18T00:42:41.243 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98486.log.gz 2024-09-18T00:42:41.243 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123626.log.gz 2024-09-18T00:42:41.243 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.99096.log.gz 2024-09-18T00:42:41.244 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67883.log.gz 2024-09-18T00:42:41.244 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64740.log.gz 2024-09-18T00:42:41.244 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.149216.log.gz 2024-09-18T00:42:41.244 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148944.log.gz 2024-09-18T00:42:41.244 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.143257.log.gz 2024-09-18T00:42:41.244 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.115602.log.gz 2024-09-18T00:42:41.244 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.150637.log.gz 2024-09-18T00:42:41.244 INFO:teuthology.orchestra.run.smithi192.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.119191.log.gz 2024-09-18T00:42:41.244 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145734.log.gz 2024-09-18T00:42:41.244 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110594.log.gz 2024-09-18T00:42:41.244 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.99201.log.gz 2024-09-18T00:42:41.244 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86997.log.gz 2024-09-18T00:42:41.244 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.148634.log.gz 2024-09-18T00:42:41.244 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.86101.log.gz 2024-09-18T00:42:41.245 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.133078.log.gz 2024-09-18T00:42:41.245 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.150372.log.gz 2024-09-18T00:42:41.245 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66858.log.gz 2024-09-18T00:42:41.245 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101591.log.gz 2024-09-18T00:42:41.245 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53390.log.gz 2024-09-18T00:42:41.245 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138290.log.gz 2024-09-18T00:42:41.245 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111183.log.gz 2024-09-18T00:42:41.245 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.92576.log.gz 2024-09-18T00:42:41.245 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136692.log.gz 2024-09-18T00:42:41.245 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102620.log.gz 2024-09-18T00:42:41.245 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.149364.log.gz 2024-09-18T00:42:41.245 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.99620.log.gz 2024-09-18T00:42:41.245 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64906.log.gz 2024-09-18T00:42:41.246 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51290.log.gz 2024-09-18T00:42:41.246 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.79504.log.gz 2024-09-18T00:42:41.246 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130389.log.gz 2024-09-18T00:42:41.246 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.150338.log.gz 2024-09-18T00:42:41.246 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66606.log.gz 89.9% 2024-09-18T00:42:41.246 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.142733.log.gz 2024-09-18T00:42:41.246 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.47857.log.gz 2024-09-18T00:42:41.246 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.59755.log.gz 2024-09-18T00:42:41.246 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48109.log.gz 2024-09-18T00:42:41.246 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.admin.45999.log.gz 2024-09-18T00:42:41.246 INFO:teuthology.orchestra.run.smithi192.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.admin.60057.log.gz 2024-09-18T00:42:41.246 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.142971.log.gz 2024-09-18T00:42:41.246 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.128694.log.gz 2024-09-18T00:42:41.246 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135263.log.gz 2024-09-18T00:42:41.247 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.128697.log.gz 2024-09-18T00:42:41.247 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.81357.log.gz 2024-09-18T00:42:41.247 INFO:teuthology.orchestra.run.smithi192.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.106980.log.gz 2024-09-18T00:42:41.247 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.152403.log.gz 2024-09-18T00:42:41.247 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.130207.log.gz 2024-09-18T00:42:41.247 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103217.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.118691.log.gz 2024-09-18T00:42:41.247 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.247 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.81355.log.gz 2024-09-18T00:42:41.247 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130736.log.gz 2024-09-18T00:42:41.247 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.119528.log.gz 2024-09-18T00:42:41.247 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.144934.log.gz 2024-09-18T00:42:41.247 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64634.log.gz 2024-09-18T00:42:41.247 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.36112.log.gz 2024-09-18T00:42:41.247 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.46520.log.gz 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.89591.log.gz 2024-09-18T00:42:41.248 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.46661.log.gz 2024-09-18T00:42:41.248 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63834.log.gz 2024-09-18T00:42:41.248 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.248 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.130257.log.gz 2024-09-18T00:42:41.248 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62868.log.gz 2024-09-18T00:42:41.248 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136060.log.gz 2024-09-18T00:42:41.248 INFO:teuthology.orchestra.run.smithi192.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.103602.log.gz 2024-09-18T00:42:41.248 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.78526.log.gz 2024-09-18T00:42:41.248 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113118.log.gz 2024-09-18T00:42:41.248 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.148171.log.gz 2024-09-18T00:42:41.248 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101383.log.gz 2024-09-18T00:42:41.248 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138642.log.gz 2024-09-18T00:42:41.248 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.100587.log.gz 2024-09-18T00:42:41.249 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123341.log.gz 2024-09-18T00:42:41.249 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43162.log.gz 2024-09-18T00:42:41.249 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66019.log.gz 2024-09-18T00:42:41.249 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42913.log.gz 2024-09-18T00:42:41.249 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134780.log.gz 2024-09-18T00:42:41.249 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65767.log.gz 2024-09-18T00:42:41.249 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.121076.log.gz 2024-09-18T00:42:41.249 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114988.log.gz 2024-09-18T00:42:41.249 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.102729.log.gz 2024-09-18T00:42:41.249 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123899.log.gz 2024-09-18T00:42:41.249 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122259.log.gz 2024-09-18T00:42:41.249 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.85975.log.gz 2024-09-18T00:42:41.249 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74732.log.gz 2024-09-18T00:42:41.250 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.135390.log.gz 89.9% 2024-09-18T00:42:41.250 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.133687.log.gz 2024-09-18T00:42:41.250 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126460.log.gz 2024-09-18T00:42:41.250 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.143590.log.gz 2024-09-18T00:42:41.250 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.77055.log.gz 2024-09-18T00:42:41.250 INFO:teuthology.orchestra.run.smithi192.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58780.log.gz 2024-09-18T00:42:41.250 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143454.log.gz 2024-09-18T00:42:41.250 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.145778.log.gz 2024-09-18T00:42:41.250 INFO:teuthology.orchestra.run.smithi192.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48332.log.gz 2024-09-18T00:42:41.250 INFO:teuthology.orchestra.run.smithi192.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.81133.log.gz 2024-09-18T00:42:41.250 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.134233.log.gz 2024-09-18T00:42:41.250 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124252.log.gz 2024-09-18T00:42:41.250 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.40852.log.gz 2024-09-18T00:42:41.250 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113545.log.gz 2024-09-18T00:42:41.251 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146008.log.gz 2024-09-18T00:42:41.251 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133124.log.gz 2024-09-18T00:42:41.251 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.96389.log.gz 2024-09-18T00:42:41.251 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% 87.3% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111601.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.118155.log.gz 2024-09-18T00:42:41.251 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.251 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.87763.log.gz 2024-09-18T00:42:41.251 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.68787.log.gz 2024-09-18T00:42:41.251 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.127220.log.gz 2024-09-18T00:42:41.251 INFO:teuthology.orchestra.run.smithi192.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47259.log.gz 2024-09-18T00:42:41.251 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.97704.log.gz 2024-09-18T00:42:41.251 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58438.log.gz 2024-09-18T00:42:41.251 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115099.log.gz 2024-09-18T00:42:41.251 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.135809.log.gz 2024-09-18T00:42:41.252 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149784.log.gz 2024-09-18T00:42:41.252 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41184.log.gz 2024-09-18T00:42:41.252 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101699.log.gz 2024-09-18T00:42:41.252 INFO:teuthology.orchestra.run.smithi192.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38919.log.gz 2024-09-18T00:42:41.252 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134865.log.gz 2024-09-18T00:42:41.252 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52257.log.gz 2024-09-18T00:42:41.252 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.79004.log.gz 2024-09-18T00:42:41.252 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70469.log.gz 2024-09-18T00:42:41.252 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135684.log.gz 2024-09-18T00:42:41.252 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.109229.log.gz 2024-09-18T00:42:41.252 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.146007.log.gz 2024-09-18T00:42:41.252 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76650.log.gz 2024-09-18T00:42:41.252 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131827.log.gz 2024-09-18T00:42:41.252 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.119153.log.gz 2024-09-18T00:42:41.253 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.149682.log.gz 2024-09-18T00:42:41.253 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.95572.log.gz 2024-09-18T00:42:41.253 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75362.log.gz 2024-09-18T00:42:41.253 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.admin.46890.log.gz 2024-09-18T00:42:41.253 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.80716.log.gz 2024-09-18T00:42:41.253 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.41276.log.gz 2024-09-18T00:42:41.253 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137279.log.gz 2024-09-18T00:42:41.253 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.96510.log.gz 2024-09-18T00:42:41.253 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.149053.log.gz 2024-09-18T00:42:41.253 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138818.log.gz 2024-09-18T00:42:41.253 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73664.log.gz 2024-09-18T00:42:41.253 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.106178.log.gz 2024-09-18T00:42:41.253 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.106406.log.gz 2024-09-18T00:42:41.254 INFO:teuthology.orchestra.run.smithi192.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.130015.log.gz 2024-09-18T00:42:41.254 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113763.log.gz 2024-09-18T00:42:41.254 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.93018.log.gz 2024-09-18T00:42:41.254 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.123183.log.gz 2024-09-18T00:42:41.254 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.67237.log.gz 2024-09-18T00:42:41.254 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148693.log.gz 2024-09-18T00:42:41.254 INFO:teuthology.orchestra.run.smithi192.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.93810.log.gz 2024-09-18T00:42:41.254 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37081.log.gz 2024-09-18T00:42:41.254 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.123604.log.gz 2024-09-18T00:42:41.254 INFO:teuthology.orchestra.run.smithi192.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59277.log.gz 2024-09-18T00:42:41.254 INFO:teuthology.orchestra.run.smithi192.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.105608.log.gz 2024-09-18T00:42:41.254 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.93174.log.gz 2024-09-18T00:42:41.254 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113533.log.gz 2024-09-18T00:42:41.254 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111438.log.gz 2024-09-18T00:42:41.255 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76104.log.gz 2024-09-18T00:42:41.255 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62281.log.gz 2024-09-18T00:42:41.255 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.138978.log.gz 2024-09-18T00:42:41.255 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.148376.log.gz 2024-09-18T00:42:41.255 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% 91.3% 89.9% -- replaced with /var/log/ceph/cluster2-client.admin.81042.log.gz 2024-09-18T00:42:41.255 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.50516.log.gz 2024-09-18T00:42:41.255 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.80257.log.gz 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.90222.log.gz 2024-09-18T00:42:41.255 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.255 INFO:teuthology.orchestra.run.smithi192.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.59907.log.gz 2024-09-18T00:42:41.255 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.127627.log.gz 2024-09-18T00:42:41.255 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.139269.log.gz 2024-09-18T00:42:41.255 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.68473.log.gz 2024-09-18T00:42:41.255 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106271.log.gz 2024-09-18T00:42:41.255 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82343.log.gz 2024-09-18T00:42:41.256 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.69101.log.gz 2024-09-18T00:42:41.256 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130108.log.gz 2024-09-18T00:42:41.256 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.admin.58982.log.gz 2024-09-18T00:42:41.256 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75212.log.gz 2024-09-18T00:42:41.256 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.99578.log.gz 2024-09-18T00:42:41.256 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124914.log.gz 2024-09-18T00:42:41.256 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137320.log.gz 2024-09-18T00:42:41.256 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.84478.log.gz 2024-09-18T00:42:41.256 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51796.log.gz 2024-09-18T00:42:41.256 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.94655.log.gz 2024-09-18T00:42:41.256 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81773.log.gz 2024-09-18T00:42:41.256 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.146241.log.gz 2024-09-18T00:42:41.256 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.46305.log.gz 2024-09-18T00:42:41.257 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.106690.log.gz 2024-09-18T00:42:41.257 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.68791.log.gz 2024-09-18T00:42:41.257 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54882.log.gz 2024-09-18T00:42:41.257 INFO:teuthology.orchestra.run.smithi192.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.103897.log.gz 2024-09-18T00:42:41.257 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.151450.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.149683.log.gz 2024-09-18T00:42:41.257 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:42:41.257 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54297.log.gz 2024-09-18T00:42:41.257 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.127329.log.gz 2024-09-18T00:42:41.257 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.103853.log.gz 2024-09-18T00:42:41.257 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.103813.log.gz 2024-09-18T00:42:41.257 INFO:teuthology.orchestra.run.smithi192.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.112272.log.gz 2024-09-18T00:42:41.257 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37848.log.gz 2024-09-18T00:42:41.257 INFO:teuthology.orchestra.run.smithi192.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.81814.log.gz 2024-09-18T00:42:41.257 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39521.log.gz 2024-09-18T00:42:41.258 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.145902.log.gz 2024-09-18T00:42:41.258 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55448.log.gz 2024-09-18T00:42:41.258 INFO:teuthology.orchestra.run.smithi192.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74961.log.gz 2024-09-18T00:42:41.258 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.115429.log.gz 2024-09-18T00:42:41.258 INFO:teuthology.orchestra.run.smithi192.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82673.log.gz 2024-09-18T00:42:41.258 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111142.log.gz 2024-09-18T00:42:41.258 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52592.log.gz 2024-09-18T00:42:41.258 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141770.log.gz 2024-09-18T00:42:41.258 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.45325.log.gz 2024-09-18T00:42:41.258 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.129150.log.gz 2024-09-18T00:42:41.258 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.127440.log.gz 2024-09-18T00:42:41.258 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.44829.log.gz 2024-09-18T00:42:41.258 INFO:teuthology.orchestra.run.smithi192.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.admin.117612.log.gz 2024-09-18T00:42:41.258 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.127313.log.gz 2024-09-18T00:42:41.259 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.128346.log.gz 2024-09-18T00:42:41.259 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.69004.log.gz 2024-09-18T00:42:41.259 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.admin.139209.log.gz 2024-09-18T00:42:41.259 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.143103.log.gz 2024-09-18T00:42:41.259 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.139797.log.gz 2024-09-18T00:42:41.259 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.81057.log.gz 2024-09-18T00:42:41.259 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91095.log.gz 2024-09-18T00:42:41.259 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.151630.log.gz 2024-09-18T00:42:41.259 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.105655.log.gz 2024-09-18T00:42:41.259 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.92533.log.gz 2024-09-18T00:42:41.259 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.78039.log.gz 2024-09-18T00:42:41.259 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.152210.log.gz 2024-09-18T00:42:41.259 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.127643.log.gz 2024-09-18T00:42:41.260 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.137030.log.gz 2024-09-18T00:42:41.260 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.139326.log.gz 2024-09-18T00:42:41.260 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106385.log.gz 2024-09-18T00:42:41.260 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.128583.log.gz 2024-09-18T00:42:41.260 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.68775.log.gz 2024-09-18T00:42:41.260 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65035.log.gz 2024-09-18T00:42:41.260 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.100586.log.gz 2024-09-18T00:42:41.260 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.104107.log.gz 2024-09-18T00:42:41.260 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.80366.log.gz 2024-09-18T00:42:41.260 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.77915.log.gz 2024-09-18T00:42:41.260 INFO:teuthology.orchestra.run.smithi192.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.148608.log.gz 2024-09-18T00:42:41.260 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.148733.log.gz 2024-09-18T00:42:41.265 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.151820.log.gz 2024-09-18T00:42:41.265 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.151465.log.gz 2024-09-18T00:42:41.265 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.127877.log.gz 2024-09-18T00:42:41.265 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72619.log.gz 2024-09-18T00:42:41.265 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.150778.log.gz 2024-09-18T00:42:41.265 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.133677.log.gz 2024-09-18T00:42:41.265 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.139212.log.gz 2024-09-18T00:42:41.265 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.140858.log.gz 2024-09-18T00:42:41.265 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129766.log.gz 2024-09-18T00:42:41.265 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.145911.log.gz 2024-09-18T00:42:41.266 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.146006.log.gz 2024-09-18T00:42:41.266 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.146217.log.gz 2024-09-18T00:42:41.266 INFO:teuthology.orchestra.run.smithi192.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46593.log.gz 2024-09-18T00:42:41.266 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.93903.log.gz 2024-09-18T00:42:41.266 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.80576.log.gz 2024-09-18T00:42:41.266 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82229.log.gz 2024-09-18T00:42:41.266 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.131154.log.gz 2024-09-18T00:42:41.266 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.45187.log.gz 2024-09-18T00:42:41.266 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.46547.log.gz 2024-09-18T00:42:41.266 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.92137.log.gz 2024-09-18T00:42:41.266 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.92938.log.gz 2024-09-18T00:42:41.266 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.56459.log.gz 2024-09-18T00:42:41.266 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-client.admin.104965.log.gz 2024-09-18T00:42:41.267 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.151906.log.gz 2024-09-18T00:42:41.267 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.56692.log.gz 2024-09-18T00:42:41.267 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.45751.log.gz 2024-09-18T00:42:41.267 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117544.log.gz 2024-09-18T00:42:41.267 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94017.log.gz 2024-09-18T00:42:41.267 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.116358.log.gz 2024-09-18T00:42:41.267 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.133250.log.gz 2024-09-18T00:42:41.267 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.105587.log.gz 2024-09-18T00:42:41.267 INFO:teuthology.orchestra.run.smithi192.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-client.mirror.148843.log.gz 2024-09-18T00:42:41.267 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.58412.log.gz 2024-09-18T00:42:41.267 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54651.log.gz 2024-09-18T00:42:41.267 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.129264.log.gz 2024-09-18T00:42:41.267 INFO:teuthology.orchestra.run.smithi192.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101468.log.gz 2024-09-18T00:42:41.267 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.152188.log.gz 2024-09-18T00:42:41.268 INFO:teuthology.orchestra.run.smithi192.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-client.mirror.37281.log.gz 2024-09-18T00:42:41.268 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.69126.log.gz 2024-09-18T00:42:41.268 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.68227.log.gz 2024-09-18T00:42:41.268 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.127729.log.gz 2024-09-18T00:42:41.268 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70603.log.gz 2024-09-18T00:42:41.268 INFO:teuthology.orchestra.run.smithi192.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.146906.log.gz 2024-09-18T00:42:41.268 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82001.log.gz 2024-09-18T00:42:41.268 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.93037.log.gz 2024-09-18T00:42:41.268 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.admin.70101.log.gz 2024-09-18T00:42:41.268 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.58184.log.gz 2024-09-18T00:42:41.271 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.58298.log.gz 2024-09-18T00:42:41.276 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.100416.log.gz 2024-09-18T00:42:41.278 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.80420.log.gz 2024-09-18T00:42:41.281 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129538.log.gz 2024-09-18T00:42:41.293 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82571.log.gz 2024-09-18T00:42:41.299 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.150544.log.gz 2024-09-18T00:42:41.299 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118342.log.gz 2024-09-18T00:42:41.302 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141884.log.gz 2024-09-18T00:42:41.306 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55281.log.gz 2024-09-18T00:42:41.308 INFO:teuthology.orchestra.run.smithi192.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42114.log.gz 2024-09-18T00:42:41.311 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.45105.log.gz 2024-09-18T00:42:41.313 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.115814.log.gz 2024-09-18T00:42:41.322 INFO:teuthology.orchestra.run.smithi192.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.149006.log.gz 2024-09-18T00:42:41.323 INFO:teuthology.orchestra.run.smithi192.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.149052.log.gz 2024-09-18T00:42:41.323 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.57471.log.gz 2024-09-18T00:42:41.339 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.130715.log.gz 2024-09-18T00:42:41.342 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.115314.log.gz 2024-09-18T00:42:41.344 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.57032.log.gz 2024-09-18T00:42:41.352 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.56682.log.gz 2024-09-18T00:42:41.364 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.141040.log.gz 2024-09-18T00:42:41.365 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47393.log.gz 2024-09-18T00:42:41.395 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.139722.log.gz 2024-09-18T00:42:41.401 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.134485.log.gz 2024-09-18T00:42:41.404 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129196.log.gz 2024-09-18T00:42:41.413 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.92004.log.gz 2024-09-18T00:42:41.416 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82457.log.gz 2024-09-18T00:42:41.436 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.103772.log.gz 2024-09-18T00:42:41.437 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.104256.log.gz 2024-09-18T00:42:41.439 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.104375.log.gz 2024-09-18T00:42:41.439 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.45445.log.gz 2024-09-18T00:42:41.450 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.139409.log.gz 2024-09-18T00:42:41.482 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58572.log.gz 2024-09-18T00:42:41.483 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58800.log.gz 2024-09-18T00:42:41.508 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.116365.log.gz 2024-09-18T00:42:41.510 INFO:teuthology.orchestra.run.smithi192.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.46776.log.gz 2024-09-18T00:42:41.512 INFO:teuthology.orchestra.run.smithi192.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.105701.log.gz 2024-09-18T00:42:41.531 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.128454.log.gz 2024-09-18T00:42:41.534 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70033.log.gz 2024-09-18T00:42:41.536 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.127515.log.gz 2024-09-18T00:42:41.537 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.140972.log.gz 2024-09-18T00:42:41.539 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.68954.log.gz 2024-09-18T00:42:41.543 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136419.log.gz 2024-09-18T00:42:41.545 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141656.log.gz 2024-09-18T00:42:41.551 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129082.log.gz 2024-09-18T00:42:41.555 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.81241.log.gz 2024-09-18T00:42:41.555 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.81727.log.gz 2024-09-18T00:42:41.558 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129424.log.gz 2024-09-18T00:42:41.560 INFO:teuthology.orchestra.run.smithi192.stderr: 94.0% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-18T00:42:41.561 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.128008.log.gz 2024-09-18T00:42:41.579 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.151253.log.gz 2024-09-18T00:42:41.583 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47164.log.gz 2024-09-18T00:42:41.590 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141086.log.gz 2024-09-18T00:42:41.595 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106499.log.gz 2024-09-18T00:42:41.600 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117430.log.gz 2024-09-18T00:42:41.601 INFO:teuthology.orchestra.run.smithi192.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.117498.log.gz 2024-09-18T00:42:41.602 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.81887.log.gz 2024-09-18T00:42:41.603 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.115577.log.gz 2024-09-18T00:42:41.606 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118456.log.gz 2024-09-18T00:42:41.609 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106043.log.gz 2024-09-18T00:42:41.621 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106157.log.gz 2024-09-18T00:42:41.628 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94359.log.gz 2024-09-18T00:42:41.633 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106613.log.gz 2024-09-18T00:42:41.639 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117886.log.gz 2024-09-18T00:42:41.671 INFO:teuthology.orchestra.run.smithi192.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.79997.log.gz 2024-09-18T00:42:41.672 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.80017.log.gz 2024-09-18T00:42:41.705 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82115.log.gz 2024-09-18T00:42:41.727 INFO:teuthology.orchestra.run.smithi192.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.mirror.38448.log.gz 2024-09-18T00:42:41.729 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59142.log.gz 2024-09-18T00:42:41.730 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58914.log.gz 2024-09-18T00:42:41.737 INFO:teuthology.orchestra.run.smithi192.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40641.log.gz 2024-09-18T00:42:41.749 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70375.log.gz 2024-09-18T00:42:41.756 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.82693.log.gz 2024-09-18T00:42:41.760 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.71059.log.gz 2024-09-18T00:42:41.772 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.94587.log.gz 2024-09-18T00:42:41.776 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58344.log.gz 2024-09-18T00:42:41.790 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47621.log.gz 2024-09-18T00:42:41.801 INFO:teuthology.orchestra.run.smithi192.stderr: 96.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59028.log.gz 2024-09-18T00:42:41.812 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141314.log.gz 2024-09-18T00:42:41.814 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141200.log.gz 2024-09-18T00:42:41.866 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70261.log.gz 2024-09-18T00:42:41.977 INFO:teuthology.orchestra.run.smithi192.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98444.log.gz 2024-09-18T00:42:41.986 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70489.log.gz 2024-09-18T00:42:41.999 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59256.log.gz 2024-09-18T00:42:42.133 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58458.log.gz 2024-09-18T00:42:42.728 INFO:teuthology.orchestra.run.smithi192.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141542.log.gz 2024-09-18T00:42:47.172 INFO:teuthology.orchestra.run.smithi096.stderr: 90.5% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-18T00:43:19.366 INFO:teuthology.orchestra.run.smithi192.stderr: 96.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35617.log.gz 2024-09-18T00:43:47.120 INFO:teuthology.orchestra.run.smithi192.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-18T00:44:48.316 INFO:teuthology.orchestra.run.smithi192.stderr: 94.7% -- replaced with /var/log/ceph/cluster1-client.mirror.0.35617.log.gz 2024-09-18T00:48:48.279 INFO:teuthology.orchestra.run.smithi192.stderr: 97.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35653.log.gz 2024-09-18T00:51:31.655 INFO:teuthology.orchestra.run.smithi192.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-18T00:52:24.171 INFO:teuthology.orchestra.run.smithi192.stderr: 94.6% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-18T00:52:26.763 INFO:teuthology.orchestra.run.smithi096.stderr: 94.1% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-18T00:53:22.669 INFO:teuthology.orchestra.run.smithi096.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-18T00:53:31.226 INFO:teuthology.orchestra.run.smithi192.stderr: 95.7% -- replaced with /var/log/ceph/cluster2-client.mirror.0.35653.log.gz 2024-09-18T00:53:39.490 INFO:teuthology.orchestra.run.smithi192.stderr: 94.5% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-18T00:53:39.492 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-09-18T00:53:39.492 INFO:teuthology.orchestra.run.smithi192.stderr:real 11m11.125s 2024-09-18T00:53:39.492 INFO:teuthology.orchestra.run.smithi192.stderr:user 3m25.476s 2024-09-18T00:53:39.492 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m16.608s 2024-09-18T00:54:14.451 INFO:teuthology.orchestra.run.smithi096.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-18T00:54:14.453 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-09-18T00:54:14.453 INFO:teuthology.orchestra.run.smithi096.stderr:real 11m46.125s 2024-09-18T00:54:14.453 INFO:teuthology.orchestra.run.smithi096.stderr:user 4m47.155s 2024-09-18T00:54:14.453 INFO:teuthology.orchestra.run.smithi096.stderr:sys 0m15.208s 2024-09-18T00:54:14.453 INFO:tasks.ceph:Archiving logs... 2024-09-18T00:54:14.454 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909591/remote/smithi096/log 2024-09-18T00:54:14.455 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T00:54:53.971 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909591/remote/smithi192/log 2024-09-18T00:54:53.972 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T00:55:24.622 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T00:55:24.660 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T00:55:24.660 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T00:55:24.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T00:55:24.765 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T00:55:24.774 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T00:55:24.774 DEBUG:teuthology.orchestra.run.smithi096:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T00:55:24.777 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T00:55:24.792 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-09-18T00:55:24.794 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-09-18T00:55:24.832 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T00:55:24.832 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-09-18T00:55:24.832 INFO:teuthology.orchestra.run.smithi096.stdout:^* hv01.front.sepia.ceph.com 3 7 377 32 -14us[ -35us] +/- 35ms 2024-09-18T00:55:24.832 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 37 -1520us[-1541us] +/- 70ms 2024-09-18T00:55:24.832 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +2056us[+2035us] +/- 90ms 2024-09-18T00:55:24.832 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T00:55:25.122 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-18T00:55:25.122 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-09-18T00:55:25.122 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv01.front.sepia.ceph.com 3 6 377 37 -53us[ -67us] +/- 35ms 2024-09-18T00:55:25.122 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 39 -1538us[-1552us] +/- 70ms 2024-09-18T00:55:25.122 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 30 +2051us[+2051us] +/- 90ms 2024-09-18T00:55:25.122 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-18T00:55:25.123 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T00:55:25.149 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T00:55:25.149 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T00:55:25.187 DEBUG:teuthology.orchestra.run.smithi096:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-18T00:55:25.282 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-18T00:55:25.384 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-18T00:55:25.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T00:55:25.414 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-18T00:55:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T00:55:25.444 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T00:55:25.453 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T00:55:25.500 INFO:teuthology.task.internal:Duration was 5438.940677 seconds 2024-09-18T00:55:25.501 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T00:55:25.555 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T00:55:25.555 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T00:55:25.558 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T00:55:25.628 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T00:55:25.670 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-18T00:55:25.952 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T00:55:25.952 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi096.front.sepia.ceph.com 2024-09-18T00:55:25.953 DEBUG:teuthology.orchestra.run.smithi096:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T00:55:25.975 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-09-18T00:55:25.976 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T00:55:25.996 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T00:55:25.996 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T00:55:26.019 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T00:55:26.063 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T00:55:26.064 DEBUG:teuthology.orchestra.run.smithi096:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T00:55:26.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T00:55:26.325 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T00:55:26.336 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T00:55:26.336 DEBUG:teuthology.orchestra.run.smithi096:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T00:55:26.404 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T00:55:26.434 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T00:55:26.445 DEBUG:teuthology.orchestra.run.smithi096:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T00:55:26.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T00:55:26.479 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = core 2024-09-18T00:55:26.514 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-09-18T00:55:26.527 DEBUG:teuthology.orchestra.run.smithi096:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T00:55:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T00:55:26.548 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T00:55:26.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T00:55:26.581 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T00:55:26.591 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T00:55:26.592 DEBUG:teuthology.misc:Transferring archived files from smithi096:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909591/remote/smithi096 2024-09-18T00:55:26.592 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T00:55:26.626 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-17_21:16:02-rbd-squid-distro-default-smithi/7909591/remote/smithi192 2024-09-18T00:55:26.626 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T00:55:26.660 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T00:55:26.660 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T00:55:26.663 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T00:55:26.710 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T00:55:26.722 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T00:55:26.722 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T00:55:26.756 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T00:55:26.757 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T00:55:26.759 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T00:55:26.772 INFO:teuthology.orchestra.run.smithi096.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 00:55 /home/ubuntu/cephtest 2024-09-18T00:55:26.774 INFO:teuthology.orchestra.run.smithi192.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 00:55 /home/ubuntu/cephtest 2024-09-18T00:55:26.776 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T00:55:26.787 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T00:55:26.857 INFO:teuthology.run:Summary data: description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_latest} workloads/compare-mirror-images-krbd} duration: 5438.9406769275665 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-18T00:55:26.858 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T00:55:26.941 INFO:teuthology.run:pass